Is it possible to upload externally collected data to Machbase Neo?


Any data consisting of numerical values can be uploaded to Machbase Neo as long as it is formatted according to its data structure.



How can external data collected be uploaded to Machbase Neo?


Machbase Neo's data formats

Machbase Neo's data formats are very simple, consisting of a table with three columns: Name, Time, and Value. Therefore, any numerical data can be uploaded to Machbase Neo by restructuring the table to have the three columns: Name, Time, and Value.

Considerations for converting data formats for Machbase Neo.

  1. Columns: To upload, the data must be transformed into a format with three columns Name, Time, and Value.

  2. Timeformat: Since Machbase Neo uses UTC time by default, it is recommended to convert the values in the time column to UTC before uploading; otherwise, the upload process can become complicated.

Convert Data Formats Example

If the data is organized in a DataFrame format, you can easily match the format through two processes.

First, Use Python's melt function to reshape the columns into the Name, Time, and Value format.

melted_df = df.melt(id_vars=['time'], var_name='name', value_name='value')

Second, Use Python's to_datetime convert the time values to UTC.


melted_df['time'] = pd.to_datetime(melted_df['time'], format='%Y-%m-%d %H:%M:%S.%f')
melted_df['time'] = melted_df['time'].dt.tz_localize('Asia/Seoul')
melted_df['time'] = melted_df['time'].apply(lambda x: int((x - epoch).total_seconds() * 1_000_000_000))

Once the data frame is restructured as shown in the image below, it will be ready for upload to Machbase Neo.

The related example code can be found on each dataset's convert.py file in the conv folder.

datahub/dataset/2024 at main · machbase/datahub
All Industrial IoT DataHub with data visualization and AI source - machbase/datahub

Finally, the data can be uploaded to Machbase Neo using the command below.


machbase-neo shell import --input ./file_name.csv.gz --compress gzip --header --method append --timeformat ns table_name

For sepecifc data migration case, see the following link : https://www.yotahub.com/tag/data-migration/

※ The command should be executed from the Machbase Neo folder location after navigating to it.