Skip to main content
Version: 1.6.0

Rest Emitter

The Rest emitter can be used to push metadata to DataHub.

​

class datahub.emitter.rest_emitter.DataHubRestEmitter(gms_server, token = None, timeout_sec = None, connect_timeout_sec = None, read_timeout_sec = None, retry_status_codes = None, retry_methods = None, retry_max_times = None, pool_connections = None, pool_maxsize = None, extra_headers = None, ca_certificate_path = None, client_certificate_path = None, disable_ssl_verification = False, openapi_ingestion = None, client_mode = None, datahub_component = None, server_config_refresh_interval = None, tcp_keepalive = None)

Bases: Closeable, Emitter

  • Parameters:
    • gms_server (str)
    • token (Optional[str])
    • timeout_sec (Optional[float])
    • connect_timeout_sec (Optional[float])
    • read_timeout_sec (Optional[float])
    • retry_status_codes (Optional[List[int]])
    • retry_methods (Optional[List[str]])
    • retry_max_times (Optional[int])
    • pool_connections (Optional[int])
    • pool_maxsize (Optional[int])
    • extra_headers (Optional[Dict[str, str]])
    • ca_certificate_path (Optional[str])
    • client_certificate_path (Optional[str])
    • disable_ssl_verification (bool)
    • openapi_ingestion (Optional[bool])
    • client_mode (Optional[ClientMode])
    • datahub_component (Optional[str])
    • server_config_refresh_interval (Optional[int])
    • tcp_keepalive (Optional[bool])

close()​

  • Return type:None

emit(item, callback=None, emit_mode=EmitMode.SYNC_PRIMARY)​

emit_mce(mce)​

emit_mcp(mcp, async_flag=None, emit_mode=EmitMode.SYNC_PRIMARY, wait_timeout=datetime.timedelta(seconds=3600))​

emit_mcps(mcps, emit_mode=EmitMode.SYNC_PRIMARY, wait_timeout=datetime.timedelta(seconds=3600))​

emit_usage(usageStats)​

fetch_server_config()​

Fetch configuration from the server if not already loaded.

  • Return type:RestServiceConfig
  • Returns: The configuration dictionary
  • Raises:ConfigurationError – If there’s an error fetching or validating the configuration

flush()​

  • Return type:None

get_server_config()​

  • Return type:dict

get_trace_status(trace, only_include_errors=False, detailed=True)​

Query the status of a traced write operation.

  • Parameters:
    • trace (TraceData) – TraceData object returned from emit_mcp() or emit_mcps()
    • only_include_errors (bool) – Only return aspects with errors (default: False)
    • detailed (bool) – Include detailed status information (default: True)
  • Return type:Optional[Dict[str, Any]]
  • Returns: Dict with URN -> aspect -> status structure, or None if server doesn’t support tracing

invalidate_config_cache()​

Manually invalidate the configuration cache.

  • Return type:None

property server_config : RestServiceConfig​

test_connection()​

  • Return type:None

to_graph()​

​

datahub.emitter.rest_emitter.DatahubRestEmitter()

alias of DataHubRestEmitter

​

class datahub.emitter.rest_emitter.EmitMode(value, names = None, *, module = None, qualname = None, type = None, start = 1, boundary = None)

Bases: ConfigEnum

ASYNC = 'ASYNC'​

ASYNC_WAIT = 'ASYNC_WAIT'​

SYNC_PRIMARY = 'SYNC_PRIMARY'​

SYNC_WAIT = 'SYNC_WAIT'​

property is_async : bool​

​

class datahub.emitter.rest_emitter.RequestsSessionConfig(**data)

Bases: ConfigModel

  • Parameters:
    • data (Any)
    • timeout (float | Tuple [float,float] | None)
    • retry_status_codes (List [int])
    • retry_methods (List [str])
    • retry_max_times (int)
    • pool_connections (int)
    • pool_maxsize (int)
    • extra_headers (Dict [str,str])
    • ca_certificate_path (str | None)
    • client_certificate_path (str | None)
    • disable_ssl_verification (bool)
    • client_mode (ClientMode | None)
    • datahub_component (str | None)
    • tcp_keepalive (bool)

build_session()​

  • Return type:Session

ca_certificate_path : Optional[str]​

client_certificate_path : Optional[str]​

client_mode : Optional[ClientMode]​

datahub_component : Optional[str]​

disable_ssl_verification : bool​

extra_headers : Dict[str, str]​

classmethod get_client_mode_from_session(session)​

Extract the ClientMode enum from a requests Session by checking the headers.

  • Parameters:session (Session) – The requests.Session object to check
  • Return type:Optional[ClientMode]
  • Returns: The corresponding ClientMode enum value if found, None otherwise

model_config : ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_input_in_errors': True, 'ignored_types': (<class 'cached_property.cached_property'>, <class 'functools.cached_property'>), 'json_schema_extra': <function _config_model_schema_extra>}​

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

pool_connections : int​

pool_maxsize : int​

retry_max_times : int​

retry_methods : List[str]​

retry_status_codes : List[int]​

tcp_keepalive : bool​

timeout : Union[float, Tuple[float, float], None]​

​

class datahub.emitter.rest_emitter.RestSinkEndpoint(value, names = None, *, module = None, qualname = None, type = None, start = 1, boundary = None)

Bases: ConfigEnum

OPENAPI = 'OPENAPI'​

RESTLI = 'RESTLI'​

​

datahub.emitter.rest_emitter.preserve_unicode_escapes(obj)

Recursively convert unicode characters back to escape sequences

  • Parameters:obj (Any)
  • Return type:Any