Quality check table
QualityCheckTable class
Views
Functions
Create a new Table
- convert to string
- remove leading and trailing whitespaces
- rename duplicates by adding _1, _2, _3, ... at the end of the name
If true, the previous rules apply plus :
- replace ' ', '-', '.' with underscores
- remove all other special characters
- remove all accents
Name | Type | Default value | Description |
---|---|---|---|
data | Union[pandas.core.frame.DataFrame, numpy.ndarray, list] | None | Data use to initilialize the table, defaults to none union[dataframe, np.ndarray, list], optional |
row_names | List | None | Set the row names for the table. must be the same size as the number of rows, defaults to none [str], optional |
column_names | List | None | Set the column names for the table. must be the same size as the number of columns, defaults to none [str], optional |
row_tags | List | None | Set the row tags for the table. must be the same size as the number of rows, defaults to none [dict[str, str]], optional |
column_tags | List | None | Set the column tags for the table. must be the same size as the number of columns, defaults to none [dict[str, str]], optional |
strict_format_header_names | bool | False | If false, the following rules are applied to headers (columns and rows) names: , optional |
Add a new column to the Dataframe.
Name | Type | Default value | Description |
---|---|---|---|
name | str | Name of the column | |
data | Union[list, pandas.core.series.Series] | None | Data for the column, must be the same length as other colums list |
index | int | None | Index for the column, if none, the column is append to the end, defaults to none , optional |
Add a tag to a column at a given index
Name | Type | Default value | Description |
---|---|---|---|
column_index | int | Index of the column | |
key | str | Key of the tag | |
value | str | Value of the tag |
Add a tag to a column by name
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | Name of the column | |
key | str | Key of the tag | |
value | str | Value of the tag |
Add a row to the Dataframe.
Name | Type | Default value | Description |
---|---|---|---|
name | str | Name of the row | |
data | Union[list, pandas.core.series.Series] | None | Data of the row list |
index | int | None | Index of the row, if none, the row is append to the end, defaults to none , optional |
Add a tag to a row at a given index
Name | Type | Default value | Description |
---|---|---|---|
row_index | int | Index of the row | |
key | str | Key of the tag | |
value | str | Value of the tag |
Add a tag to a row by name
Name | Type | Default value | Description |
---|---|---|---|
row_name | str | Name of the row | |
key | str | Key of the tag | |
value | str | Value of the tag |
Add technical information on the resource. Technical info are useful to set additional information on the resource.
Name | Type | Default value | Description |
---|---|---|---|
technical_info | TechnicalInfo | Technical information to add (key, value) |
Checks if a column with the given name exists in the Table and raises an exception if it doesn't.
:raises Exception: If the column doesn't exist.
Name | Type | Default value | Description |
---|---|---|---|
name | str | The name of the column to check. | |
case_sensitive | bool | True | Whether the check should be case sensitive. defaults to true. |
You can redefine this method to define custom logic to check this resource. If there is a problem with the resource, return a string that define the error, otherwise return None This method is called on output resources of a task. If there is an error returned, the task will be set to error and next proceses will not be run. It is also call when uploading a resource (usually for files or folder), if there is an error returned, the resource will not be uploaded
Union[str, NoneType] |
Checks if a row with the given name exists in the Table and raises an exception if it doesn't.
:raises Exception: If the row doesn't exist.
Name | Type | Default value | Description |
---|---|---|---|
name | str | The name of the row to check. | |
case_sensitive | bool | True | Whether the check should be case sensitive. defaults to true. |
Clone the resource to create a new instance with a new id. It copies the RFields.
ResourceType |
Checks if a column with the given name exists in the Table.
Name | Type | Default value | Description |
---|---|---|---|
name | str | The name of the column to check. | |
case_sensitive | bool | True | Whether the check should be case sensitive. defaults to true. |
bool |
Copy column tags from source_table to self matching by index.
Name | Type | Default value | Description |
---|---|---|---|
source_table | Table | Source table to copy tags from | |
from_index | int | None | Start index of the columns to copy, defaults to none , optional |
to_index | int | None | End index of the columns to copy, defaults to none , optional |
Copy column tags from source_table to self matching by name.
Name | Type | Default value | Description |
---|---|---|---|
source_table | Table | Source table to copy tags from |
Copy row tag from source_table to self matching by index
Name | Type | Default value | Description |
---|---|---|---|
source_table | Table | Source table to copy tags from | |
from_index | int | None | Start index of the rows to copy, defaults to none , optional |
to_index | int | None | End index of the rows to copy, defaults to none , optional |
Copy row tag from source_table to self matching by name
Name | Type | Default value | Description |
---|---|---|---|
source_table | Table | Source table to copy tags from |
Create a new table from a dataframe and a meta
Name | Type | Default value | Description |
---|---|---|---|
dataframe | DataFrame | The dataframe | |
row_tags | List | The list of row tags [dict[str, str]] | |
column_tags | List | The list of column tags [dict[str, str]] |
Table |
Create a sub Table based on a subset Dataframe of this original table filtered by columns It copies the tags of this table into the new table based on column names that matched between filtered_df and this dataframe.
Name | Type | Default value | Description |
---|---|---|---|
filtered_df | DataFrame | The filtered dataframe |
Table |
Create a sub Table based on a subset Dataframe of this original table filtered by rows. It copies the tags of this table into the new table based on row names that matched between filtered_df and this dataframe.
Name | Type | Default value | Description |
---|---|---|---|
filtered_df | DataFrame | The filtered dataframe |
Table |
Check if the table is equal to another table. It compares the data, row tags and column tags.
Name | Type | Default value | Description |
---|---|---|---|
o | object | The other table |
bool |
Create a new row and fill it with the values of the tag of each column.
If none, tag key is used as name, defaults to None
Name | Type | Default value | Description |
---|---|---|---|
tag_key | str | ||
new_row_name | str | None | Name of the new row that will contains tag values. , optional |
Create a new tag for each row and fill it with the values of the provided column
If none, column name is used as key, defaults to None
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | Name of the column to extract | |
new_tag_key | str | None | Key of the new tag that will contains column values. , optional |
delete_column | bool | False | If true, delete the column after the extraction, defaults to false , optional |
Create a new columns and fill it with the values of the tag of each row
If none, tag key is used as name, defaults to None
Name | Type | Default value | Description |
---|---|---|---|
tag_key | str | ||
new_column_name | str | None | Name of the new column that will contains tag values. , optional |
Create a new tag for each column and fill it with the values of the row.
Name | Type | Default value | Description |
---|---|---|---|
row_name | str | Name of the row to extract values from | |
new_tag_key | str | None | Key of the new tag that will contains row values. if none, row name is used as key, defaults to none , optional |
delete_row | bool | False | If true, delete the row after the extraction, defaults to false , optional |
Filter out table columns matching a list of names, return a new table
Name | Type | Default value | Description |
---|---|---|---|
filters | List | The list of names [dataframefiltername] |
Table |
Filter out table rows matching a list of names, return a new table
Name | Type | Default value | Description |
---|---|---|---|
filters | List | The list of names [dataframefiltername] |
Table |
Filter out table rows or columns matching a list of tags and return a new table. The row or column that matches the tags are removed.
Example of search tags are:
tags = [ {"key1": "value1"} ]
to filter out rows or columns having a tag{"key1": "value1"}
tags = [ {"key1": "value1", "key2": "value2"} ]
to filter out rows or columns having tags{"key1": "value1"} AND {"key2": "value2"}
tags = [ {"key1": "value1"}, {"key2": "value2"} ]
to filter out rows or columns having tags{"key1": "value1"} OR {"key2": "value2"}
tags = [ {"key1": "value1", "key2": "value2"}, {"key3": "value3"} ]
to filter out rows or columns having tags({"key1": "value1"} AND {"key2": "value2"}) OR {"key2": "value2"}
- AND and OR logics can further be combined to perform complex selects
Name | Type | Default value | Description |
---|---|---|---|
axis | Literal[0, 1, 'index', 'columns'] | Axis to filter out the tags axistype | |
tags | List | The of tags [dict] |
Table |
Generates a column name that is unique in the Dataframe base on name. If the column name doesn't exist, return name, otherwise return name_1 or name_2, ... Only the name is returned, the column is not added to the Dataframe.
Name | Type | Default value | Description |
---|---|---|---|
name | str | Name of the column |
str |
Get the available tags for each column.
Dict |
Get the available tags for each row.
Dict |
Get the value of a cell at a given coordonate (row, column)
Name | Type | Default value | Description |
---|---|---|---|
row_index | int | Index of the row | |
column_index | int | Index of the column |
Any |
Returns a column with the given name as a DataFrame.
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | The name of the column. | |
skip_nan | bool | False | Whether to skip nan values. defaults to false. |
DataFrame |
Returns the data of a column with the given name.
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | The name of the column. | |
skip_nan | bool | False | Whether to skip nan values. defaults to false. |
List |
Get the index of a column from its name
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | Name of the column |
int |
Get the info of a column by name
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | Name of the column |
TableColumnInfo |
Get the column names
Name | Type | Default value | Description |
---|---|---|---|
from_index | int | None | Start index of the columns to retrieve, defaults to none , optional |
to_index | int | None | End index of the columns to retrieve, defaults to none , optional |
List |
Function to retrieve the column names based on column indexes
Name | Type | Default value | Description |
---|---|---|---|
indexes | List | List of column indexes [int] |
List |
Get the tags of multiple columns by index
Name | Type | Default value | Description |
---|---|---|---|
from_index | int | None | Start index of the columns to retrieve, defaults to none , optional |
to_index | int | None | End index of the columns to retrieve, to_index is included, defaults to none , optional |
none_if_empty | bool | False | If true, return none if no tags are found, defaults to false , optional |
List |
Get the tags of a column at a given index
Name | Type | Default value | Description |
---|---|---|---|
column_index | int | Index of the column |
Dict |
Get the tags of a column by name
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | Name of the column |
Dict |
Get the type of a column
Name | Type | Default value | Description |
---|---|---|---|
column_name | Any | Name of the column str |
TableColumnType |
Get the info of multiple columns by index
Name | Type | Default value | Description |
---|---|---|---|
from_index | int | None | Start index of the columns to retrieve, defaults to none , optional |
to_index | int | None | End index of the columns to retrieve, defaults to none , optional |
List |
DataFrame |
You can redefine this method to set a name of the resource. When saving the resource the name will be saved automatically This can be useful to distinguish this resource from another one or to search for the resource
str |
Returns the data of a row with the given name.
Name | Type | Default value | Description |
---|---|---|---|
row_name | str | The name of the row. | |
skip_na | bool | False | Whether to skip nan values. defaults to false. |
List |
Get the index of a row from its name. Raise an exception if the row doesn't exist
Name | Type | Default value | Description |
---|---|---|---|
row_name | str | Name of the row |
int |
Get the info of a row by name
Name | Type | Default value | Description |
---|---|---|---|
row_name | str | Name of the row |
TableHeaderInfo |
Get the row names of the table by index
Name | Type | Default value | Description |
---|---|---|---|
from_index | int | None | Start index of the rows to retrieve, defaults to none , optional |
to_index | int | None | End index of the rows to retrieve, defaults to none , optional |
List |
Function to retrieve the row names based on row indexes
Name | Type | Default value | Description |
---|---|---|---|
indexes | List | List of row indexes [int] |
List |
Get the tags of a row by name
Name | Type | Default value | Description |
---|---|---|---|
row_name | str | Name of the row |
Dict |
Get the tags of multiple rows by index
Name | Type | Default value | Description |
---|---|---|---|
from_index | int | None | Start index of the rows to retrieve, defaults to none , optional |
to_index | int | None | End index of the rows to retrieve, to_index is included, defaults to none , optional |
none_if_empty | bool | False | If true, return none if no tags are found, defaults to false , optional |
List |
Get the tags of a row at a given index
Name | Type | Default value | Description |
---|---|---|---|
row_index | int | Index of the row |
Dict |
Get the info of multiple rows by index
Name | Type | Default value | Description |
---|---|---|---|
from_index | int | None | Start index of the rows to retrieve, defaults to none , optional |
to_index | int | None | End index of the rows to retrieve, defaults to none , optional |
List |
Get the tags of a given axis
Name | Type | Default value | Description |
---|---|---|---|
axis | Literal[0, 1, 'index', 'columns'] | Axis to retrieve the tags axistype |
List |
Get the technical information of the resource
Name | Type | Default value | Description |
---|---|---|---|
key | str | Key of the technical information |
TechnicalInfo |
Returns the first n rows for the columns ant targets.
Name | Type | Default value | Description |
---|---|---|---|
nrows | int | 5 | Number of rows |
DataFrame |
Call infer_objects on the underlying dataframe, it modifies the table dataframe.
Table |
This can be overwritten to perform custom initialization of the resource. This method is called just after the init (constructor) of the resource. The default values of RFields are set before this method is called.
Remove a column from the Dataframe.
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | Name of the column |
Remove a row from the Dataframe.
Name | Type | Default value | Description |
---|---|---|---|
row_name | str | Name of the row to remove |
Checks if a row with the given name exists in the Table.
Name | Type | Default value | Description |
---|---|---|---|
name | str | The name of the row to check. | |
case_sensitive | bool | True | Whether the check should be case sensitive. defaults to true. |
bool |
Select table columns matching a list of indexes, return a new table
Name | Type | Default value | Description |
---|---|---|---|
indexes | List | The list of indexes [int] |
Table |
Select table columns matching a list of names, return a new table
Name | Type | Default value | Description |
---|---|---|---|
filters | List | The list of names [dataframefiltername] |
Table |
Select table columns matching a list of tags
Example of search tags are:
tags = [ {"key1": "value1"} ]
to select columns having a tag{"key1": "value1"}
tags = [ {"key1": "value1", "key2": "value2"} ]
to select columns having tags{"key1": "value1"} AND {"key2": "value2"}
tags = [ {"key1": "value1"}, {"key2": "value2"} ]
to select columns having tags{"key1": "value1"} OR {"key2": "value2"}
tags = [ {"key1": "value1", "key2": "value2"}, {"key3": "value3"} ]
to select columns having tags({"key1": "value1"} AND {"key2": "value2"}) OR {"key2": "value2"}
- AND and OR logics can further be combined to perform complex selects
Name | Type | Default value | Description |
---|---|---|---|
tags | List | The of tags [dict] |
Table |
Create a new table from coords. It does not includes the to_row_id and to_column_id
Name | Type | Default value | Description |
---|---|---|---|
from_row_id | int | Start row index | |
from_column_id | int | Start column index | |
to_row_id | int | End row index | |
to_column_id | int | End column index |
Table |
Select table rows matching a list of indexes, return a new table
Name | Type | Default value | Description |
---|---|---|---|
indexes | List | The list of indexes [int] |
Table |
Select table rows matching a list of names, return a new table
Name | Type | Default value | Description |
---|---|---|---|
filters | List | The list of names [dataframefiltername] |
Table |
Select table rows matching a list of tags
Example of search tags are:
tags = [ {"key1": "value1"} ]
to select rows having a tag{"key1": "value1"}
tags = [ {"key1": "value1", "key2": "value2"} ]
to select rows having tags{"key1": "value1"} AND {"key2": "value2"}
tags = [ {"key1": "value1"}, {"key2": "value2"} ]
to select rows having tags{"key1": "value1"} OR {"key2": "value2"}
tags = [ {"key1": "value1", "key2": "value2"}, {"key3": "value3"} ]
to select rows having tags({"key1": "value1"} AND {"key2": "value2"}) OR {"key2": "value2"}
- AND and OR logics can further be combined to perform complex selects
Name | Type | Default value | Description |
---|---|---|---|
tags | List | The of tags [dict] |
Table |
Select table rows or columns matching a list of tags and return a new table
Example of search tags are:
tags = [ {"key1": "value1"} ]
to select rows or columns having a tag{"key1": "value1"}
tags = [ {"key1": "value1", "key2": "value2"} ]
to select rows or columns having tags{"key1": "value1"} AND {"key2": "value2"}
tags = [ {"key1": "value1"}, {"key2": "value2"} ]
to select rows or columns having tags{"key1": "value1"} OR {"key2": "value2"}
tags = [ {"key1": "value1", "key2": "value2"}, {"key3": "value3"} ]
to select rows or columns having tags({"key1": "value1"} AND {"key2": "value2"}) OR {"key2": "value2"}
- AND and OR logics can further be combined to perform complex selects
Name | Type | Default value | Description |
---|---|---|---|
axis | Literal[0, 1, 'index', 'columns'] | Axis to select the tags axistype | |
tags | List | The of tags [dict] |
Table |
Select only numeric columns, return a new table
if drop_na = 'any', then drop columns where any values are nan (similar to `DataFrame.drop_na(how=all|any)`)
Name | Type | Default value | Description |
---|---|---|---|
drop_na | Literal['all', 'any'] | all | If drop_na = 'all', then drops columns where all values are nan (similar to `dataframe.drop_na(how=all|any)`) literal['all', 'any'] |
Table |
Set the names of all columns
Name | Type | Default value | Description |
---|---|---|---|
column_names | List | List of column names list |
Set the tags of all columns, the length of the list must be equal to the number of columns
Name | Type | Default value | Description |
---|---|---|---|
tags | List | List of tags [dict[str, str]] |
Set the names of all rows
Name | Type | Default value | Description |
---|---|---|---|
row_names | List | List of row names, must be the same length number of rows list |
Set the tags of all rows, the length of the list must be equal to the number of rows
Name | Type | Default value | Description |
---|---|---|---|
tags | List | List of tags [dict[str, str]] |
Set the value of a cell at a given coordonate (row, column)
Name | Type | Default value | Description |
---|---|---|---|
row_index | int | Index of the row | |
column_index | int | Index of the column | |
value | Any | Value to set |
Set the name of a column at a given index
Name | Type | Default value | Description |
---|---|---|---|
current_name | str | Current name of the column | |
new_name | str | New name of the column |
Set the tags of a column at a given index
Name | Type | Default value | Description |
---|---|---|---|
column_index | int | Index of the column | |
tags | Dict | Tags to set [str, str] |
Set the tags of a column by name
Name | Type | Default value | Description |
---|---|---|---|
column_name | str | Name of the column | |
tags | Dict | Tags to set [str, str] |
Name | Type | Default value | Description |
---|---|---|---|
comments | str | '' |
Update the name of a row
Name | Type | Default value | Description |
---|---|---|---|
current_name | Any | Current name of the row str | |
new_name | str | New name of the row |
Set the tags of a row at a given index
Name | Type | Default value | Description |
---|---|---|---|
row_index | int | Index of the row | |
tags | Dict | Tags to set [str, str] |
Set the tags of a row by name
Name | Type | Default value | Description |
---|---|---|---|
row_name | str | Name of the row | |
tags | Dict | Tags to set [str, str] |
Returns the last n rows for the columns ant targets.
Name | Type | Default value | Description |
---|---|---|---|
nrows | int | 5 | Number of rows |
DataFrame |
Returns the table as a csv string.
str |
Returns the table as a pandas dataframe.
DataFrame |
Returns the table as a json string.
dict |
Returns the table as a list of lists.
List |
Returns the table as a numpy array.
ndarray |
Transpose the table, it returnes a new Table, the original table is not modified.
Name | Type | Default value | Description |
---|---|---|---|
infer_objects | bool | False |
Table |