[Generated by Task Expert Agent]

Load and process BiolectorXT data from raw measurements and metadata files.

Overview

This task integrates BiolectorXT data from raw measurements and metadata to create a comprehensive dataset for microplate analysis. It handles data parsing, well labeling, quality control visualization, and generates statistics about data completeness.

Input Files Required

1. Raw Data Table (raw_data)

Table containing raw measurement data from BiolectorXT with columns:

• Well: Well identifier (e.g., "A01", "B02")
• Filterset: Filter/channel name (e.g., "Biomass", "pH", "DO")
• Time: Measurement time in seconds
• Cal: Calibrated measurement value
• Additional metadata columns may be present

2. Metadata Folder (folder_metadata)

Folder containing JSON metadata file(s) ending with 'BXT.json':

• Channels: List of measurement channels/filters
• Microplate: Well configuration
  • CultivationLabels: Wells used for cultivation
  • ReservoirLabels: Wells used as reservoirs
• Layout: Well label descriptions
  • CultivationLabelDescriptionsMap: Descriptions for cultivation wells
  • ReservoirLabelDescriptionsMap: Descriptions for reservoir wells
• Comment: Experiment comment/description
• Name: Experiment name
• UserName: User who created the experiment
• LastModifiedAt: Last modification date

3. Plate Layout (Optional) (plate_layout)

JSONDict containing custom well labels and additional metadata:

• Keys: Well identifiers (e.g., "A1", "A01")
• Values: Dict with label key and optional additional metadata
• Overrides metadata labels if provided

Processing Steps

1. Metadata Extraction: Reads BXT.json file from metadata folder
2. Data Parsing: Transforms raw data from long to wide format
   • Groups by Filterset to separate different measurement channels
   • Creates intermediate tables with wells as columns
   • Handles both microfluidics (C01-F08) and standard (A01-F08) layouts
3. Data Restructuring: Pivots data to create one table per well
   • Each table contains time column and all measurement channels
   • Filters out wells with no data (all NaN)
4. Well Labeling: Merges labels from metadata and optional plate layout
5. Quality Control: Tracks well data availability
   • Cultivation wells
   • Reservoir wells
   • Labeled wells
6. Tagging: Adds comprehensive tags from metadata
   • Batch (experiment/plate name) and sample (well ID)
   • Experiment name, comment, user, date
   • Raw data source
   • Well-specific labels and metadata
7. Statistics: Generates metadata summary table with well counts

Outputs

1. Parsed Data Tables (parsed_data_tables)

A ResourceSet containing one Table per well (batch/sample combination):

• Table Name: Well identifier (e.g., "A01", "B02", "C03")
• Columns:
  • Temps_en_h: Time in hours
  • One column per measurement channel (e.g., "Biomass", "pH", "pO2", "DO")
• Resource Tags:
  • batch: Experiment/plate name (from metadata Name field)
  • sample: Well identifier (e.g., "A01")
  • label: Well label/description (if available)
  • comment: Experiment comment
  • name: Experiment name
  • user_name: User who created the experiment
  • date: Last modification date
  • raw_data: Raw data table name
  • biolector_download: Download tag (if present in raw data)
  • Additional custom metadata from plate_layout

2. Venn Diagram (venn_diagram) - Optional

A PlotlyResource containing an interactive Venn diagram showing:

• 3 Overlapping Circles: Cultivation, Reservoir, Labeled
• Circle Labels: Show count of wells in each category
• Center Label: Shows count of fully characterized wells (all 3 categories)
• Color Coding:
  • Blue: Cultivation wells
  • Green: Reservoir wells
  • Purple: Labeled wells

3. Metadata Summary Table (metadata_summary) - Optional

A Table containing experiment-level statistics:

• Columns:
  • metric: Metric name
  • value: Metric value
• Metrics:
  • Total channels/filters
  • Total wells
  • Cultivation wells count
  • Reservoir wells count
  • Labeled wells count
  • Experiment name
  • User name
  • Comment
  • Last modified date

Data Quality

Well Type Detection

The task automatically identifies:

• Cultivation wells: From metadata CultivationLabels
• Reservoir wells: From metadata ReservoirLabels
• Labeled wells: Wells with non-empty labels from metadata or plate layout

Microfluidics Detection

Automatically detects microfluidics mode:

• Checks if "A01" is present in well identifiers
• If not present: Microfluidics mode (C01-F08 wells)
• If present: Standard mode (A01-F08 wells)

Use Cases

1. Quality Control: Use Venn diagram to assess well labeling completeness
2. Data Exploration: Browse parsed data with proper well labels
3. Batch Processing: Process multiple experiments with consistent structure
4. Dashboard Preparation: Provides clean, tagged data ready for visualization
5. Downstream Analysis: Standardized format for filtering, analysis tasks

Example Workflow

[BiolectorXT Download] ──┬──> [Raw Table]
                         └──> [Metadata Folder]
                                  │
                                  │
[Plate Layout JSON] ─────────────┼──> BiolectorXTLoadData ──┬──> [Well Tables] ──> Filter/Analysis
                                                              ├──> [Venn Diagram] ──> QC Report
                                                              └──> [Metadata Summary] ──> Stats

Example Output Structure

For a plate with 3 wells (A01, A02, B01) and 2 measurements (Biomass, pH):

ResourceSet containing 3 tables:

Table "A01":
  Temps_en_h | Biomass | pH
  0.0        | 0.123   | 7.2
  0.5        | 0.156   | 7.1
  1.0        | 0.198   | 7.0

Table "A02":
  Temps_en_h | Biomass | pH
  0.0        | 0.115   | 7.3
  0.5        | 0.142   | 7.2
  ...

Notes

• All JSON files must be UTF-8 encoded
• Metadata file must end with 'BXT.json'
• Well identifiers are normalized (e.g., "A1" → "A01")
• Wells with no data (all NaN) are excluded from the output
• Time is provided in hours (Temps_en_h)
• Each table represents one well with all its measurements
• Output format is compatible with filtering and analysis tasks
• Plate layout overrides metadata labels when provided
• Tags include batch (experiment name) and sample (well ID) for easy filtering

Comparison with BiolectorXTDataParser

This task differs from BiolectorXTDataParser:

• Different output structure: One table per well instead of one table per channel
• Added features:
  • Venn diagram for data availability visualization
  • Metadata summary table with experiment statistics
  • Batch/sample tagging for consistent data organization
• Enhanced documentation with usage examples
• Consistent naming with other load tasks (e.g., FermentalgLoadData)