Service logs support via Generic Extensions

[gberche-orange]

As a service user:

• in order to use services managed by a 3rd party provider but still:
  • get out of band service usage logs such as RDBMS slow query logs
  • get event notifications from the service (such as RDS Event notification like availability, backup, failover, failure, patching...)
• I need to receive logs from the services I subscribed through my preferred platform, and possibly my apps to consume these logs in an automated way.

As a broker author, in order to provide logs to service users and bound apps, I need a standard way of exposing metrics to platforms, in particular, I need:

• a secured, authenticated way to provide logs associated to each service instance and possibly service binding
• a portable way to provide logs across platforms:
  • available to both public services and private on-prem services (i.e. the service broker is'nt required to acquire privileged platform permissions to expose logs to end-users)

Related pointers:

• Heroku add-on partner api logs params for add-ons to provide logs that will be sent to app's log stream

[n3wscott]

Have you tried implementing log-drain type bindings? https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#log-drain

The broker author has to support this type of binding.

[gberche-orange]

@n3wscott sorry for late response. I understand the log-drain binding type enables a service to collect logs from an application through a syslog endpoint (e.g. application logs search/archive/alerting). This issue is more about services pushing service related logs to applications.

[mattmcneeney]

As @gberche-orange, syslog drains allow service instances to receive logs from applications rather than the other way around.

Do we think this issue would be solved by the extensions proposal (#431), whereby service instances could host an endpoint that applications could poll to fetch logs?

[gberche-orange]

I realize I misworded the issue, so I updated it to reflect that service logs would need to be consumed by app developers first, and then potentially by app themselves:

I need to receive logs from the services I subscribed through my preferred platform, and possibly my apps to consume these logs in an automated way.

@mattmcneeney

Do we think this issue would be solved by the extensions proposal (#431), whereby service instances could host an endpoint that applications could poll to fetch logs?

In the case of cloudfoundry, consumption of application logs by developers is standardized through the loggregator support (with both CF CLI and streaming to log management systems UX).

I believe it would make sense that service instances logs get consumed by app developers through the same UX. This may require platforms to poll a service provided endpoint, and stream the service instances logs through the log pipeline.

Additionally, service brokers could return log polling endpoint in their binding responses so that apps can poll them, in order to automate service instance logs processing. Maybe OSB support for log endpoint in binding responses would help keep a consistent experience across services.

[mattmcneeney]

Thanks for the clarification @gberche-orange! That makes sense to me. I think this feature request can be broken down into two parts:

1. Having a standardised way to fetch logs from service instances.
   This is the part that I believe will be solved by Generic Extensions (actions) #431 and having a community standard for what fetching logs looks like (probably an OpenAPI document).
2. Having platforms automatically discover when logs can be fetched from service instances (again, probably through Generic Extensions (actions) #431 and a community standard for logging) and present that to developers (platform-specific UX feature).

Does that make sense? I believe we should track part (1) in this issue (developing the OpenAPI standard for fetching logs from service instances), and part (2) will be tracked by platform authors in their respective forums.

[gberche-orange]

thanks @mattmcneeney . Sounds good to me.

For clarity, would part (2) also cover how platforms would be presenting log endpoints to applications, in similar way than service binding credentials are injected to applications by platforms ?

[mattmcneeney]

Yes I believe so!

[gberche-orange]

For the CF community, the related epic in SAPI backlog, if this may help follow CF exchanges on the topic.

[mattmcneeney]

Closing this. Work on the generic extensions work can be tracked in #670 !

[mattmcneeney]

Reopened, updated title and added the blocked label so that we tackle this after the generic extensions (#670) work is done.