Streaming from your Cameras

Introduction

Smarter AI Stream SDK ensures smooth and high quality streaming experience between application and camera. You can stream your cameras from anywhere using 3G/4G/WiFi. Smarter AI Stream SDK for Android manages the streaming logic and underlying operations which manages up to four concurrent streaming at your mobile device. There are some interesting features here. It can notify you network connectivity during streaming. Using this SDK, you can record audio and video of a stream. It can also record based on motion sensor event. It can also provide AI data of detected object for vision camera.

Concept

Streaming in Smarter AI platforms requires to login into the system and the camera has to be connected to internet and then it has to be assigned under your Access Control List. After the camera is onboarded and you got permission to view the streaming from mobile app then you will have option to start, stop, pause and resume operations.

Understanding the process

Smarter AI Stream SDK provide you a handsome number of operations to stream audio and video into multiple mobile app at a time. It also supports ABS (Adaptive Bitrate Streaming). But for availaing all those options from camera app you just need to follow couple of steps and then sthe stream sdk will communicate with the mobile app smartly.

Developing Camera App: Streaming

Necessary Resources

Stream.h
Available in include folder. It contains the APIs related to stream.

Access.h
Available in include folder. It contains the APIs related to access.

libacstream.so
Available in lib folder.

libacaccess.so
Available in lib folder.

Dependency

Stream library internally uses

libacconnect.so
libvision.so

Access library internally uses

libsensor.so

Steps:

Introduction

Implementing Steps

1. Initialize Access library

In previous section how to Initialize Access Library using builder-pattern. This is a pre-requisite of creating stream instance.

2. Setting Access Callback methods

In Callback registration section there is brief discussion how to use the callback methods to get notified asynchronously.

    access->registerEndpointResetCallback([this](const types::Endpoint endpointId) {
        // Take necessary actions like releasing access and stream, restarting pairing etc
    });  

    access->registerEndpointInfoChangeCallback([](const types::ChangedInfo &info, const types::Endpoint endpointId, const std::string &changedData) {
        // Receive the changes in endpoint
    });

Endpoint
ChangeInfo is an enum which is present in AccessTypes.h which holds the types of changes in endpoint.

enum class ChangedInfo {
    Name,
    ImageURL,
    Ipv4Address,
    Ipv6Address,
    SDP,
    RecordingRelaySDP,
    Unknown ///Never use, only for internal purpose
};

3. Initialize Stream library

In previous section we have shown how to Initialize stream library using builder-pattern.

Streaming

4. Setting Callback methods

you have to register some callback methods with the stream instance. The callback methods are hit by stream library to notify about an event. Following is sample code for registering some stream event with callbacks.

    stream->onStreamReceive([this](const com::anyconnect::stream::EndPointUUID &toEndPoint, const com::anyconnect::stream::MediaType &MediaType) -> bool {
        // Stream received
        // Take nececssary Actions
        return true;
    });

    stream->onStreamStop([this](const com::anyconnect::stream::EndPointUUID &fromEndPoint, bool status) -> bool {
        // Stream stopped for that endpointId
        // Take ncessary Actions
        return true;
    });

    stream->onStreamStart([this](const com::anyconnect::stream::EndPointUUID &fromEndPoint, const std::string &connectionType) -> bool {
        // Stream started for that endpointId
        // Take necessary Action
    return true;
    });

    stream->onStreamConnectivityChange([this](const EndPointUUID &fromEndpoint, const StreamTransitionState streamTransitionState, const std::string &connectionType) -> void {
        // Stream connectivity changed for that endpointId

    });

    stream->onRecordingCompleted([this](const com::anyconnect::stream::EndPointUUID &fromEndPoint, const int& status) {
        //Recording completed for `fromEndPointId`
    });

5. Configuring Stream Instance

Enable Adaptive Bitrate Stream
It will change the bit rate depending on bandwidth and cpu usage.

    stream->setQoS(true);

Set Media Sources
Use setInputSources callback method to set the Video Source

    //Media source description. Format for Video is "name type width height fps bitrate", Audio format is "name codec samplerate bitrate(kbit/s)" Example Video source: "vid_1 video 1280 720 30 600000". For Audio source: "aud_1 opus 16 48". You may provide multiple audio/video source if appropriate
    std::string audioInputSources("aud_1 audio opus 16 48")
    streamRet = stream->setInputSources(MediaType::AUDIO, audioInputSources, [this](const std::vector<std::string> &inputSourceList, const std::vector<InputSourceStatus> &status) {
        //check if audio sources setup is sucessful or not
    });
    std::string videoInputSources("vid_1 video 1280 720 30 600000");
    streamRet = stream->setInputSources(MediaType::VIDEO, videoInputSources, [this](const std::vector<std::string> &inputSourceList, const std::vector<InputSourceStatus> &status) {
        //check if video sources setup is sucessful or not
    });

6. Start Streaming

When the stream instance is setup with necessary configuration and callback, it is time to get ready stream instance for streaming. The following sample code shows how to get stream instance ready for streaming. Internally stream module will prepare connect module, vision module, recording module and others.

    com::anyconnect::stream::StreamRet strRet = stream->start();
    if (strRet == com::anyconnect::stream::StreamRet::OK) {
        // Stream Started Succesfully
    } else {
        // Stream Failed to Start
        return false;
    }

Stream Recording

How Streaming Works

Smarter AI Stream Sdk has option of stream recording. Whenever a event is captured by event module it hits back access callback method and from camera app the event is further received by another callback.

Stream Recording

Set Recording buffer time

By setting the buffer time you set stream sdk to record stream for additional buffer time so that total event recording time = buffer time before the event + recording after the event (post buffer time). In the following sample code RECORDING_BUFFER_TIME seconds is buffer time before the event and RECORDING_DURATION seconds is total recording time.

    stream->setRecordingBuffer(RECORDING_BUFFER_TIME, RECORDING_DURATION);

Necessary APIs

There are some APIs in Stream.h that are meant to be used for the recording operations. Those are as follows:

    ///
    /// Register callback function to receive system events generated by
    /// event plugin library.
    ///
    /// \param[in] callback - Asynchronous callback handler that shall be invoked
    ///                       when event fired by event library.
    ///
    /// \return - AccessRet::OK if successful, other values on failure.
    ///
    virtual AccessRet registerSystemEventListener(SystemEventCallback callback) = 0;

    /// onStartRecordingHandler will tell if recording is started or not.
    ///
    /// \param[in] fromEndPoint stream source endpoint UUID
    /// \param[in] mediaTypeList Selected media types (audio/video)
    /// \param[in] eventTrigger true if the recording initiated for an event.
    /// \param[in] handler function pointer for the callback.
    ///
    /// Note that, this API will be used by both the camera and the application
    /// to start the recording. When invoked from the camera the fromEndPoint will
    /// be empty.
    ///
    virtual bool startRecording(const EndPointUUID &fromEndPoint, time_t referenceTime, time_t actionTime, time_t duration, const std::vector<MediaType> &mediaTypeList, const std::vector<std::string> &sourceList, bool eventTrigger, onStartRecordingHandler handler) = 0;

    ///
    /// Set callback to get notified when a recording completed.
    ///
    /// \param[in] handler function pointer which will be called when a recording has completed.
    ///
    virtual void onRecordingCompleted(onStopRecordingHandler handler) = 0;


    ///
    /// Request to stop the recording.
    ///
    /// \param[in] fromEndPoint stream source endpoint UUID
    /// \param[in] mediaTypeList selected media types
    /// \param[in] handler function pointer for the callback.
    ///
    virtual void stopRecording(const EndPointUUID &fromEndPoint, const std::vector<MediaType> &mediaTypeList, onStopRecordingHandler handler) = 0;

Callback Handlers

Both of these APIs take callback handlers as input and calls back asynchronously with recording status. The callback handlers are as follows:

    ///
    /// Handler function for start recording.
    ///
    /// \param[in] fromEndPoint stream source endpoint UUID
    /// \param[in] status requested recording status
    ///
    typedef std::function<bool(const EndPointUUID &fromEndPoint, const int &status)> onStartRecordingHandler;

    ///
    /// Handler function for stop recording.
    ///
    /// \param[in] fromEndPoint stream source endpoint UUID
    /// \param[in] status recording status
    ///
    typedef std::function<void(const EndPointUUID &fromEndPoint, const int &status)> onStopRecordingHandler;

Please check the following sample codes to understand the process better.

Listening to Event

Following is an example how to register for events from access instance. When an event is found then the callback will be hitted and you can call to start recording

    access->registerSystemEventListener([this](const std::string& eventType, const com::anyconnect::access::types::Properties eventData, const std::string& timestamp) {
        // if event is found then
        // call to start recording
    });

Start Recording

Once the event is found call to start streaming like below

    std::vector<std::string> audioInputSources;
    stream->getInputSources(MediaType::AUDIO, audioInputSources);

    std::vector<std::string> videoInputSources;
    stream->getInputSources(MediaType::VIDEO, videoInputSources);

    auto endpointId = (EndPointUUID) 1234;
    std::vector<MediaType> mediaTypes = {MediaType::AUDIO, MediaType::VIDEO};

    stream->startRecording(endpointId, eventTime, 0, 0, mediaTypes, inputSources, true, [this, endpointId, eventTime, mediaTypes, inputSources, evData](const EndPointUUID &fromEndPoint, const int &status) -> bool {
        if (status) {
            // recording started successfully
        } else {
            // recording failed to start
        }
        return true;
    });

Recording Completed

You should set a callback handler to know whether the recording is completed:

    stream->onRecordingCompleted([this](const com::anyconnect::stream::EndPointUUID &fromEndPoint, const int& status) {
        //process recording completed.
        //call for stop recording
    });

Stop Recording

When the recording is completed then you should call the asynchronous method stopRecording. Please look at the sample code below:

    std::promise<void> taskRec;
    stream->stopRecording(epId, mediaTypes, [this, &taskRec](const EndPointUUID &fromEndPoint, const int& status) {
        // recording stopped successfully
        taskRec.set_value();
    });
    taskRec.get_future().get();