etl::ETLService Class Reference

This class is responsible for continuously extracting data from a p2p node, and writing that data to the databases. More...

#include <ETLService.hpp>

Inheritance diagram for etl::ETLService:

Collaboration diagram for etl::ETLService:

Public Member Functions
	ETLService (util::async::AnyExecutionContext ctx, std::reference_wrapper< util::config::ClioConfigDefinition const > config, std::shared_ptr< data::BackendInterface > backend, std::shared_ptr< LoadBalancerInterface > balancer, std::shared_ptr< NetworkValidatedLedgersInterface > ledgers, std::shared_ptr< LedgerPublisherInterface > publisher, std::shared_ptr< CacheLoaderInterface > cacheLoader, std::shared_ptr< CacheUpdaterInterface > cacheUpdater, std::shared_ptr< ExtractorInterface > extractor, std::shared_ptr< LoaderInterface > loader, std::shared_ptr< InitialLoadObserverInterface > initialLoadObserver, std::shared_ptr< TaskManagerProviderInterface > taskManagerProvider, std::shared_ptr< MonitorProviderInterface > monitorProvider, std::shared_ptr< SystemState > state)
	Create an instance of ETLService.
void	run () override
	Start all components to run ETL service.
void	stop () override
	Stop the ETL service.
boost::json::object	getInfo () const override
	Get state of ETL as a JSON object.
bool	isAmendmentBlocked () const override
	Check for the amendment blocked state.
bool	isCorruptionDetected () const override
	Check whether Clio detected DB corruptions.
std::optional< ETLState >	getETLState () const override
	Get the etl nodes' state.
std::uint32_t	lastCloseAgeSeconds () const override
	Get time passed since last ledger close, in seconds.

Static Public Member Functions
static std::shared_ptr< ETLServiceInterface >	makeETLService (util::config::ClioConfigDefinition const &config, util::async::AnyExecutionContext ctx, std::shared_ptr< BackendInterface > backend, std::shared_ptr< feed::SubscriptionManagerInterface > subscriptions, std::shared_ptr< LoadBalancerInterface > balancer, std::shared_ptr< NetworkValidatedLedgersInterface > ledgers)
	A factory function to spawn new ETLService instances.

Detailed Description

This class is responsible for continuously extracting data from a p2p node, and writing that data to the databases.

Usually, multiple different processes share access to the same network accessible databases, in which case only one such process is performing ETL and writing to the database. The other processes simply monitor the database for new ledgers, and publish those ledgers to the various subscription streams. If a monitoring process determines that the ETL writer has failed (no new ledgers written for some time), the process will attempt to become the ETL writer.

If there are multiple monitoring processes that try to become the ETL writer at the same time, one will win out, and the others will fall back to monitoring/publishing. In this sense, this class dynamically transitions from monitoring to writing and from writing to monitoring, based on the activity of other processes running on different machines.

Constructor & Destructor Documentation

ETLService()

etl::ETLService::ETLService	(	util::async::AnyExecutionContext	ctx,
		std::reference_wrapper< util::config::ClioConfigDefinition const >	config,
		std::shared_ptr< data::BackendInterface >	backend,
		std::shared_ptr< LoadBalancerInterface >	balancer,
		std::shared_ptr< NetworkValidatedLedgersInterface >	ledgers,
		std::shared_ptr< LedgerPublisherInterface >	publisher,
		std::shared_ptr< CacheLoaderInterface >	cacheLoader,
		std::shared_ptr< CacheUpdaterInterface >	cacheUpdater,
		std::shared_ptr< ExtractorInterface >	extractor,
		std::shared_ptr< LoaderInterface >	loader,
		std::shared_ptr< InitialLoadObserverInterface >	initialLoadObserver,
		std::shared_ptr< TaskManagerProviderInterface >	taskManagerProvider,
		std::shared_ptr< MonitorProviderInterface >	monitorProvider,
		std::shared_ptr< SystemState >	state )

Create an instance of ETLService.

Parameters

ctx	The execution context for asynchronous operations
config	The Clio configuration definition
backend	Interface to the backend database
balancer	Load balancer for distributing work
ledgers	Interface for accessing network validated ledgers
publisher	Interface for publishing ledger data
cacheLoader	Interface for loading cache data
cacheUpdater	Interface for updating cache data
extractor	The extractor to use
loader	Interface for loading data
initialLoadObserver	The observer for initial data loading
taskManagerProvider	The provider of the task manager instance
monitorProvider	The provider of the monitor instance
state	System state tracking object

Member Function Documentation

getETLState()

std::optional< ETLState > etl::ETLService::getETLState ( ) const

overridevirtual

Get the etl nodes' state.

Returns

The etl nodes' state, nullopt if etl nodes are not connected

Implements etl::ETLServiceInterface.

getInfo()

boost::json::object etl::ETLService::getInfo ( ) const

overridevirtual

Get state of ETL as a JSON object.

Returns

The state of ETL as a JSON object

Implements etl::ETLServiceInterface.

isAmendmentBlocked()

bool etl::ETLService::isAmendmentBlocked ( ) const

overridevirtual

Check for the amendment blocked state.

Returns

true if currently amendment blocked; false otherwise

Implements etl::ETLServiceInterface.

isCorruptionDetected()

bool etl::ETLService::isCorruptionDetected ( ) const

overridevirtual

Check whether Clio detected DB corruptions.

Returns

true if corruption of DB was detected and cache was stopped.

Implements etl::ETLServiceInterface.

lastCloseAgeSeconds()

std::uint32_t etl::ETLService::lastCloseAgeSeconds ( ) const

overridevirtual

Get time passed since last ledger close, in seconds.

Returns

Time passed since last ledger close

Implements etl::ETLServiceInterface.

makeETLService()

std::shared_ptr< ETLServiceInterface > etl::ETLService::makeETLService ( util::config::ClioConfigDefinition const & config, util::async::AnyExecutionContext ctx, std::shared_ptr< BackendInterface > backend, std::shared_ptr< feed::SubscriptionManagerInterface > subscriptions, std::shared_ptr< LoadBalancerInterface > balancer, std::shared_ptr< NetworkValidatedLedgersInterface > ledgers )

static

A factory function to spawn new ETLService instances.

Creates and runs the ETL service.

Parameters

config	The configuration to use
ctx	Execution context for asynchronous operations
backend	BackendInterface implementation
subscriptions	Subscription manager
balancer	Load balancer to use
ledgers	The network validated ledgers datastructure

Returns

A shared pointer to a new instance of ETLService

run()

void etl::ETLService::run ( )

overridevirtual

Start all components to run ETL service.

Implements etl::ETLServiceInterface.

stop()

void etl::ETLService::stop ( )

overridevirtual

Stop the ETL service.

Note

This method blocks until the ETL service has stopped.

Implements etl::ETLServiceInterface.

---

The documentation for this class was generated from the following files:

• /__w/clio/clio/src/etl/ETLService.hpp
• /__w/clio/clio/src/etl/ETLService.cpp