Bilkent University
Department of Computer EngineeringSenior Design Project
Low Level Design Report
DRIvision
Students
Musab Gelişgen
Ebru Kerem
Yusuf Dalva
Kuluhan Binici
Zeynep Sonkaya
Supervisor
Asst. Prof. Hamdi Dibeklioğlu
Jury Members
Asst. Prof. Dr. A. Ercüment Çiçek
Assoc. Prof. Dr. Çiğdem Gündüz Demir
February 17, 2020
This report is submitted to the Computer Engineering Department of Bilkent University in
partial fulfillment of the requirements of the Senior Design Project course CS491/2.
1. Introduction 3
1.1. Object Design Trade-offs 4
1.1.1. Response Time vs Computing Performance 4
1.1.2. Development and Sustainability Cost vs Performance 4
1.1.3. Compatibility vs Programmability 4
1.1.4. Flexibility vs Usability 4
1.2. Interface Documentation Guidelines 5
1.3. Engineering Standards 5
1.4. Definitions, acronyms, and abbreviations 5
2. Packages 6
2.1. Presentation Tier Packages 8
2.1.1. Activity 8
2.1.2. Fragment 9
2.1.3. Service 9
2.1.4. Floating View 10
2.2. Logic Tier Packages 10
2.2.1. Cloud/Phone Computation Manager 11
2.2.2. App Controller 12
2.2.3. Image Taker 13
2.2.4. Models 14
2.3. Data Tier Packages 14
2.3.1. AWS - Cloud Computing Server 15
2.3.2. Google Drive Server 16
2.3.3. Video Log 17
2.3.4. Selected Features 18
3. Class Interfaces 19
3.1. Presentation Tier Classes 19
3.1.1. Activity Package Classes 19
3.1.2. Fragment Package Classes 19
3.1.3. Service Package Classes 19
3.1.4. Floating View Package Classes 20
3.2. Logic Tier Classes 22
3.2.1. App Controller Package Classes 22
3.2.2. ImageTaker Package Classes 25
3.2.3. CloudComputation Package Classes 25
3.2.4. Models Package Classes 26
3.3. Data Tier Classe 27
3.3.1. AWS Cloud Computing Server Package Classes 27
3.3.2. Google Drive Server Package Classes 27
3.3.3. VideoLog Package Classes 28
3.3.4. SelectedFeatures Package Classes 29
4. Glossary 30
1.
Introduction
Transportation is one of the daily needs we have in a global world. People need to change their location to continue their life. Along with public transport, vehicles are the most common way of transportation. From the perspective of people in traffic, we paid attention to make an improvement and creating some impact on the personal vehicle side seemed to make more sense. From the perspective of personal vehicles, the main issue that creates danger in traffic was the human factor. Human decision making may not make the optimal choice every time but it can be supported with tools, which is the main idea behind this project.
Due to this possible error in the decision making process of the drivers, our team came up with the idea of developing an application that can warn the driver of possible dangers and improve the driving experience with the features of the application that is proposed in the next sections of this report. The main motivation of the project depends on the detection of possible dangers, uncomfortable situations and then to warn the driver or make some adjustments independent than the vehicle in the application (like preventing a fogless view to the driver).
In the development of the idea, we realized that the features that we are going to supply in this application are also supplied with the cars that are produced with the most recent technology. However, from the client's point of view, these improvements in the car’s standard equipment cost a significant amount compared to the price of the car itself according to our market research. At this point, the application is going to be differentiated from similar products from the price point by providing a cheap alternative to the additional hardware that is provided by the car manufacturers.
From the aspect of the qualifications that are provided by our application, we are going to offer an all-in-one package for driving assisting systems (like lane identification traffic sign recognition, etc). The contents of the proposed package are going to be explained in the latter sections. With the advantage that the price point of the application brings, as a team our product is going to make a significant impact. In order to provide these features to the drivers, we are going to use several machine learning techniques, that make our application be able to benefit from the latest prediction techniques to provide insights with a low error rate. In this report, you will find the design trade-offs in several subsections, followed with the high-level explanations of the packages that will be put into 3-tier structure of the system. Then, the class interfaces will be given and explained in detail.
1.1. Object Design Trade-offs
1.1.1.
Response Time vs Computing Performance
In the application, the user is able to run multiple models for assistance. Due to limitations of the software devices in terms of computation power, running all of the models would affect performance poorly. Due to this fact the application takes aid from Amazon Web Services(AWS) as a cloud computing service in order to compute the necessary outputs for the application. However, since the communication between the physical device (mobile device that the application will be run) and the cloud server that the model is going to be
running will take time, this adds an additional computational time for the requested result by the application. In this regard, the object design also tries to deal with the balance between the application performance (in terms of response time) and the overall performance of the mobile device considering the computational burden coming from the applications that would be running concurrently with our application.
1.1.2.
Development and Sustainability Cost vs Performance
As mentioned in the previous section, the application uses cloud computing services (AWS) for additional computation power for running the related models based on computer vision principles. Since the cloud services offered by Amazon come with different configurations in terms of computation power requested the application is going to be effected in terms of performance according to the configuration selected for computing. In order not to create any additional cost because of using the cloud services, the base configurations is selected (m2.t2.medium). The primary financial resource for the cloud services will be the student credits provided by Amazon. In order not to create any additional cost for the project, the decrease in performance is considered as a trade-off.
1.1.3.
Compatibility vs Programmability
As mentioned in the previous stages of the project (with the reports that are previously submitted), the application is offered only for Android platform. The low-level design proposed in this report is prepared considering an Android application that is programmed mainly by using Java. Due to limited compatibility of the application, only the devices that operate on the Android version that operate Android 8.0 (Oreo) minimum. This decision has been made in order to use some of the core functionalities provided by the operating system. The application is not designed as a multi-platform application in order to provide a complete application performing well according to the specifications decided at the earlier stages of the project. Also in order to implement some of the features (using the applicatioın and the navigation app at the same time) the operating system version Android 8.0 is mandatory. This fact has a negative effect on the compatibility of the application but has a positive effect on the programmability of the application.
1.1.4.
Flexibility vs Usability
By the nature of the application proposed, the application promises assistance to the driver while driving. Considering the fact that the navigation applications are being used while driving a major decision for the application has been made and the design has been done in a way that compatibility with the navigation applications will be maintained. For our application the design has been made in such a way that the main functionality of the application is being able to be used in background and positioning the necessary warnings and additional visuals will be done to the screen over another application. By providing this flexibility there is a possibility that the usability of the application will decrease due to the fact that the standard configuration will conflict with the user interface of the application running other than ours. However, since we are planning to provide a general purpose application, we
decided to make this trade-off and optimize the positioning of the visuals provided in the best way.
1.2. Interface Documentation Guidelines
In this report, we use the following convention to write a low-level design: ● Package name: This indicates which class belongs to which package
● Class name The specific class name which contains the following attributes and methods
● Description: A brief explanation of class
● Attributes: "+" indicates that the attribute is public and "-" indicates it is private. The type of the attribute is followed by ":".
● Methods "+" indicates that the method is accessible from outside of the class and "-" indicates that it can be used only within the class while “#” is for the protected package access. Furthermore, their return types are specified after the ":" sign.
1.3. Engineering Standards
In this report, two different engineering standards has been followed. For the diagrams given in the report, which form low-level design diagram altogether is constructed by following UML design principles[1]. For the citations made the IEEE citation standards has been followed[2]. As a standard in the design, the mediator design pattern has been use where the main control of flow is performed by manager classes[3].
1.4. Definitions, acronyms, and abbreviations
Some terms and abbreviations are defined below in order to ease the understanding of the content in the letter sections.
Machine Learning Model: It can be defined as a black box system that embodies a mathematical representation of a real-world process.
AWS: Is the abbreviation for Amazon Web Services, which is a widely-used cloud platform that provides tools and resources for software developers.
Detection: This term refers to the concept of object detection, which can be defined as detecting instances of objects from particular classes in images.
API: Is the abbreviation for the term Application Programming Interface. The term can be defined as an interface to an application component that is designed to ease the usage of such components by other developers than the owner.
TensorFlowLite: Is the file format to store trained machine learning models for usage in Android applications.
Data Transfer Object (DTO): Is used to provide communication between remote processes while passing data.
2.
Packages
This section contains the package diagram of the DRIVision system.
In our choice of architectural design, we wanted to have maximum possible modularity, thus divided the system into packages and possible classes in a way to have modularity. We believe our design will have high coherence and low coupling. Three tier architecture pattern is used in our design with the intention of allowing any of the three tiers to be upgraded or replaced independently in response to changes in requirements or technology.
Our three-tier architecture which contains tiers the user interface (presentation), functional process logic, data tiers. These tiers are developed and maintained as independent modules, most often on the same platform but different layers. Our three-tier also can be called a three-layer structure with hierarchical relation between layers.
2.1. Presentation Tier Packages
This is the top most level of the application. The presentation tier displays information related to detected possible dangers during the travel, accessed data from third party map’s API such as speed limit or view of road when fog/rain removed. Also this tier is used to interact with the user via voice recognition during order taking and giving verbal outputs.
Presentation tier puts out the results to the screen visually and annonces some of them verbally. Results it outputs are obtained from other tiers in the network while the presentation layer communicates with other tiers. In simple terms, it is a layer which users can access directly via the DRIvision application.
In the design of this tier, we will use adaptable user interface which does not disturb the driver’s use of mobile phone during the use of DRIvision. The user can view his/her desired navigation application or other desired applications. DRIvision interface that will be implemented in the presentation tier is designed accordingly.
2.1.1.
Activity
Figure 2.Package Diagram of Activity
The activity package contains the DRIVisionActivity class which is used to instantiate the application. The DRIVisionActivity class contains fragments that are loaded on the screen when the app is opened. The fragments consist of FloatingWindows which are explained in Section 2.1.4.
2.1.2.
Fragment
Figure 3.Package Diagram of Fragment
This package contains the FloatingViewControlFragment class which is used to show FloatingWindows on top of the screen. In addition, this class interacts with the service package which serves as intermediate manager of the system.
2.1.3.
Service
Figure 4.Package Diagram of Service
This package contains the FloatingViewService class which has a FloatingViewManager instance. By this instance, a FloatingView can be controlled via its functions. This class serves as an intermediate controller/manager in the app. What a FloatingView should display is controlled through this class.
2.1.4.
Floating View
Figure 5.Package Diagram of FloatingView
This package contains the main presentation elements of the app which are FloatingViews. A Floating view is an element which can be displayed on the phone on top of any app, i.e, it is always on the screen regardless of what other app is running. These elements are mostly small rectangles and circles while they serve as display and control tools. Since DRIVision app is an assistance app for the drivers, this gives the drivers the opportunity to use DRIVision while other navigation applications are running such as Google Maps and Yandex Navi.
The Floating Manager class controls the state, positioning of each FloatingView. FullScreenObserverView is used to provide the connection between the Android ecosystem and the application itself. There are a couple of FloatingWindow types one of which is the TrafficSignView that shows the most recent speed limit red by the camera. CameraView is to show the filtered foggy/rainy images. AppControlView is used to maintain overall app visibility and settings while the SettingsView provides the drivers with the features to enable.
2.2.1.
Cloud/Phone Computation Manager
Figure 6.Package Diagram of CloudComputation
The CloudComputation package consists of a single class that is responsible for providing cloud computation for the deep learning utilization required by some features of the application. The ComputationManager class is responsible for using the AWS cloud API to send the captured image frames to the appropriate models on the cloud for processing, based on user preferences, and receiving the inference results from the cloud. After these steps, it interacts with the ResponseManager to give appropriate responses to the user or to update the UI.
2.2.2.
App Controller
Figure 7.Package Diagram of AppController
The AppController package consists of 5 classes that manages the flow of events that are happening while the application is running. The ResponseManager class is responsible for giving appropriate reactions to the user and updating the UI based on the model inference
results it receives from the ModelManager. For instance if the output of the traffic sign recognition model indicates that the speed sign is different than the lastly updated one, the ResponseManager interacts with the presentation layer objects to change the traffic sign displayed on the screen. The FeatureSelectionManager class is responsible for interacting with the data layer to retrieve the current feature selections by the user and updating them upon change of user preference. The speed limit class is responsible for both retrieving the current speed limits during the journey and storing them. To provide such functionality, it interacts with the ModelManager and the navigation API to get the speed limits at the current location. It contains an inner class called MyLocationListener which retrieves the current location of the device from GPS. LogRecordingManager class is responsible for recording the image frames retrieved by the ImageTaker, when the user commands so upon detecting faulty traffic behavior. It interacts with the data layer to store the video logs locally and also with the Google Drive API to store them in Google Drive. VoiceProcessing class is used to detect speech and recognize when the user gives a command to start or stop the video log recording. Upon recognition of such command, it interacts with LogRecordingManager for taking the requested action.
2.2.3.
Image Taker
Figure 8.Package Diagram of ImageTaker
The ImageTaker package consists of 2 classes that retrieves camera inputs as image frames. The CameraController class is responsible for recording image frames at a certain rate, delegating them to the ModelManager for model inference and storing consecutive frames within a CameraVideo object, when the user commands the application to record a faulty traffic behavior. The CameraVideo class is responsible for storing the series of recorded consecutive frames.
2.2.4.
Models
Figure 9.Package Diagram of Models
The Models package consists of 2 classes that provide deep learning utilization for the relevant application features. The ModelManager class is responsible for keeping track of the active models, which varies based on user preference, transferring the captured image frames to these models for inference and delegating the received outputs to the ResponseManager for giving appropriate responses to the user based on model inference results. The Model class is a wrapper for the Tensorflow's Classifier class and it is responsible for providing model inference. The Classifier it contains can store various network definitions such as object detectors or rain filterers based on the model file that is used for the initialization. Therefore, as the inference results can be of different data types, the output of the inference function is an Object. In order to distinguish the actual output type/structure that we receive after inference, so that we can make use of it for giving appropriate responses, we also added the name of the model object as a string attribute.
2.3. Data Tier Packages
The data tier includes the data access layer that encapsulates the persistence mechanisms and exposes the data. The data access layers (Servers in the Data Tier) provide an API to the logic tier that exposes methods of managing the stored data without exposing or creating dependencies on the data storage mechanisms. These so called server packages include subsystems that use API of third party software.
Figure 10. Package Diagram for Data Layer
As with the separation of any tier, there are costs for implementation and often costs to performance in exchange for improved scalability and maintainability. Each of the packages are constructed in a way that the optimization of the different aspects of the application (such as scalability and maintainability) can be done without affecting the contents of the other packages. In this regard, the packages are organized in the following way:
- Package for interacting with cloud computing server (2.3.1) - Package for interacting with the cloud storage (2.3.2) - Package for maintaining video logs (2.3.3)
- Package for maintaining the user preferences in terms of the models that are going to be used and information about the models (2.3.4)
2.3.1.
AWS - Cloud Computing Server
Figure 11. Package Diagram for AWS - Cloud Computing Server package
The application is going to use the AWS Cloud in order to provide the results of some of the models to the user. Amazon Web Services offer the functionality of running precompiled
models in a web server which gives results to the application by using HTTP requests to the web server that is being used[4]. In order to handle the requests to the web server for some specific model the controller inside this package is going to be used. The web server is going to return an object which is the expected output of the model as a prediction. Due to the difference between the outputs of the models the generic type Object has been selected. After receiving the result from the web server, the necessary operations will be done by using the classes in the logic tier.
This package contains one class as a member which is CloudPredictor. The brief explanation of the class is given below:
CloudPredictor: This class instantiates a runtime object in order to connect to AWS Cloud Server and with a model specified by the path name, the necessary output is requested from the server. This class has a single method along with the constructor of the class. The method establishes connection with the server and sends the inputted series of image (or video) and gives the output from the web server
2.3.2.
Google Drive Server
Figure 12. Package Diagram for Google Drive Server package
For our application, the users have the option of saving their video logs to a Google Drive server which is authenticated. In order to perform this type of functionality, the application needs to communicate and be able to upload files to the Google Drive Server. For this purpose, there is a specific package which is composed of one class. This class is mainly responsible for managing credentials and making necessary HTTP requests by using Google
Drive API[5]. This class is connected by using VideoLogManager class which is going to be explained in the latter section. The class contained in this package is explained briefly below:
DriveManager: This class is responsible for all of the Google Drive related operations performed in the application. The functionality of this class can be divided into two subcategories which are authentication and file operations. For the authentication part, the manager performs the creation of the credential file (which is stored locally) and performs authentication before uploading the file[6]. In terms of file operations, the creation of the file in the true format and then uploading this file to the Google Drive Server is performed.
2.3.3.
Video Log
Figure 13. Package Diagram for VideoLog package
The only feature that the user is going to save data to the application or to a connected cloud account is taking video logs at the application. As this is the only way that the user can modify the memory usage in the application, a specific package is dedicated to that purpose. Following the common pattern followed in the application there is a class for the Video that is going to be saved and there is a manager for controlling the saving operation. In the application the user will have the option of saving the video to the local storage and to the cloud storage which is the main reason why there is a specific manager for video logs. Here the VideoLog class is thought as a DTO again here the saving operation is going to be performed by its manager. The brief descriptions of the classes included are given below:
VideoLog: Containing the necessary data for video record. This class also generates a specific file name and sets the necessary attributes for recording the video. For a specific log, there is a directory, a file name and an Intent object for the generation of the video file. As Android operating system saves files in a video format by using an Intent object, there is an Intent object as a private attribute of this class.
VideoLogManager: The storage operation of a Video log is performed by this manager class. According to the users preference on using a Google Drive account or not, the storage of a video file is performed by this object. By this class the connection to the Google Drive Server package is established and the file storage operation is done by using this package.
2.3.4.
Selected Features
Figure 14. Package Diagram for SelectedFeatures package
The selected features package handles the feature selection which is done by the user and saves the features saved to the application. The main purpose of this package is providing the ability to the cloud server and the device itself the easiness of accessing to the models that are offered to the user and being used according to the selection done by the end-user. In order to manage the preference of the user and the model running operation, this package contains 3 classes. These classes are ModelData, SelectedFeatures and FeatureSelectionManager. Their respective functionalities are given below as brief descriptions:
ModelData: This class is responsible for storing the paths of a definite model which is going to be used at the stage of computation. Respectively, this class contains the name of the model, the path for running it locally and the path for running the class in cloud. The class returns the respective model path according to the decision of the application. This is created as a data transfer object (DTO) just in order to access the data quickly about a respective model.
SelectedFeatures: The features that are specified by the user effects the model files that are going to be used. According to the preferences of the user, a respective list of ModelData objects will be created and then stored inside the application accordingly. Then the information about a particular model that is selected by the user can be accessed easily.
FeatureSelectionManager: With this class, the updates on the user preferences can be
reflected on the application instantly. This class controls the features selected by the user and operates the writing operations of the features metadata.
3.
Class Interfaces
3.1.
Presentation Tier Classes
3.1.1.
Activity Package Classes
Class: DRIVisionActivity
This class initiates the application and loads the fragments to run.
Attributes:
Functions:
protected onCreate(Bundle savedInstanceState) takes action depending on the
device’s Android version and instantiates Fragments.
3.1.2.
Fragment Package Classes
Class: FloatingViewControlFragment
This class triggers the floating views to be displayed on the screen.
Attributes:
Functions:
public onCreateView(ViewGroup container, Bundle savedInstanceState) adds
floating views to the view group and passes the information to the bundle, returns the view.
private showFloatingView(Context: context) creates the floating views to be displayed. private static startFloatingViewService(Activity activity) wrapper function to create the FloatingViewService.
3.1.3.
Service Package Classes
Class: FloatingViewService
This class serves as an intermediate controller/manager in the app.
FloatingViewManager floatingViewManager is the instance of the Manager class for communication with FloatingViews.
Functions:
public onStartCommand(Intent intent) contains come rules to be applied to the service when it instantiates
private loadOptions(DisplayMetrics metrics) takes the metrics passed and delivers this
information to the FloatingView. The metrics contain information such as what to display on the floating view.
private static createNotification(Context context) gets the information about the
Floating View which is being interacted with.
3.1.4.
Floating View Package Classes
Class: FloatingView
This class is the main element of the app which is displayed on the screen.
Attributes:
private WindowManager manager WindowManager.LayoutParams params
int visibility: is used to control Floating View visibility
Functions:
private dispatchTouchEvent(MotionEvent event) defines what should happen in a certain type of interaction happens with the user. This function is called in onTouchEvent function which adds the action listeners to the element.
private addMovement(MotionEvent event) is called from the dispatcher function when
the user drags the element.
private updateViewConfiguration(DisplayMetrics metrics)updates the information on a FloatingView
Class: FloatingViewManager
This class controls the main element of the app which is displayed on the screen.
Attributes:
private List<FloatingViewList> list FullScreenOberverView view
public addViewToScreen(FloatingView view) is called when the requested floating
views need to be added to the screen.
private removeView(FloatingView view) is called when the requested floating views
need to be removed from the screen
private removeAllViews() removes all the existing FloatingViews.
Class: FullScreenObserverView
This class acts as an intermediate class between screen display and the DRIVision app.
Attributes:
WindowManager.LayoutParams params
Functions:
protected onAttachedToWindow defines the actions to be taken when an element is added to the screen.
protected onDetachedFromWindow defines the actions to be taken when an element is removed from the screen.
Class: AppControlView
This class is to provide elements to control/shutdown the DRIVision app.
Attributes:
Icon minimize Icon shutDown int state
Functions:
public onMinimize(MotionEvent event) is used to hide all the components when clicked. public onMaximize(MotionEvent event) is used to show all the components when
clicked.
public onShutDown(MotionEvent event) is used to quit the app.
Class: SettingsView
This class is to provide elements to enable/disable features..
WindowManager.LayoutParams params
Functions:
public showSettings(MotionEvent event) opens a window showing the features to enable/disable.
public toggleFeature(MotionEvent event, Option option) is used to toggle a feature’s state.
3.2.
Logic Tier Classes
3.2.1.
App Controller Package Classes
App controller package controls the app, binds packages with each other and controls models with combining them. The package organizes the warnings and output types. It redirects the other classes to get desired results.
Class: ResponseManager
This class manages models usages and get desired results to be used by the presentation tier.
Attributes:
Functions:
private getCollisionAlarm() returns the alarm number after interacting with the results of head posture and obstacle detection models. When there is no detected collision it returns 0, if collision possible according to head posture and lane detection result it returns 1.
private getFogRainRemovedImage() returns the fog and rain filtered out version of the current camera image to display in the screen according to model.
private getDistractionAlarm() returns the alarm number after interacting with the results
of head posture and lane detection models. When there is no distracted driver detected it returns 0, if driver distracted possible according to head posture and lane detection result it returns 1.
private getSpeedBumphole() returns integer where 0 means there is no speed bump or hole, 1 means there is speed bump but no hole, 2 means there is hole but no speed bump and 3 means there are both speed bump and pothole in the image.
private getTrafficSign() returns the id number of the traffic sign detected according to the traffic sign detection model.
public getModelResponser(Map<String str,Object obj> modelResultMap) calls all the class methods according to the featureSelectionManager’s getSelectedFeatures() method takes the results of result of models as Object and string pair. String indicated which model’s result Object is. Presentation tier will use this method.
Class: SpeedLimit
This class gets the speed limit from API and gives to the system.
Attributes:
● ArrayList<String> speedLimit
● BroadcastReceiver minuteUpdateReceiver ● MapOverpassAdapter overApiAdapter
Functions:
public fetchMaxSpeed(LatLngBounds bound): returns string arraylist. This method
takes the given coordinate and sends a query to Overpass API. Then, it fills the array of string which consists of the speed limit of that given locations.
public locationUpdater(): This method returns boolean values. While the driver moves, in order to get speed limits, location has to be updated. If the location is updated regularly, it returns true, otherwise it returns false.
private displayGPSStatus(): returns true or false. If GPS is open and we can retrieve the location information, it returns true or it returns false.
public startMinuteUpdater(LatLngBounds bound): In addition to location change, the system also gets value periodically. This method controls that functionality.
Class: LogRecordingManager
This class controls the video taking feature. It manages where to put the video and also when to start and to finish the video.
Attributes:
Functions:
public saveVideoToDrive(): returns boolean values. If the Internet is available and the user gives access to the system to reach his/her drive, the videos will be saved into the drive with that method.
public startVideoTaking(): returns true or false. When the user presses the log recording button, this method will take control and will activate video recording.
public saveVideoLocally(): returns boolean values. If the user does not give access to the application to reach his/her account or the user uses the application offline, the video will be saved in the loca and this method controls that functionality.
public stopVideoTaking(): returns true or false. When the user presses the stop button, this method will take control and will deactivate the video recording.
public moveNotUploadedToDrive(): returns boolean values. When the Internet is
connected, this method manages to upload videos which are selected to upload drive but they are not uploaded yet.
Class: FeatureSelectionManager
The application gives the user option to change features based on his/her needs. This class controls selection of features.
Attributes:
Functions:
public updateSelectedFeatures(boolean[] features): returns true or false. If the user changes his/her choices, the system will be updated with the help of this method.
public getSelectedFeatures(boolean[] features): returns boolean values. This method
helps to get selected features and calls them.
Class: MyLocationListener
This class controls the location information. It works with SpeedLimit class to get the speed limit of that location.
Attributes:
Functions:
public onProviderDisabled(String provider): When the given provider is disabled, it handles the situation and gives error.
public onLocationChanged(Location loc): This method handles the change in the
location and updates the information of the location.
public onProviderEnabled(String provider):When the given provider is enabled, it gives command to the needed classes.
Class: VoiceProcessing
This class helps to convert the voice commands to the text to activate specific features such as log recording.
Attributes:
Functions:
public convertVoiceToText(File voice): This method takes the voice file and converts it into the string and gives it to the program.
3.2.2.
ImageTaker Package Classes
Class: CameraController
Controls Image creation process from Camera and creates Videos by latest Images
Attributes:
Functions:
public generateImageToFeedModels(): returns a Image taken by Camera of mobile public generateCameraVideo(): generates a CameraVideo by latest 10 seconds
Class: CameraVideo
Storage Object for Images to be used to generate Videos from taken Images to be used as LogRecords
Attributes:
public ArrayList<Image> lastImages Image array for the latest 10 seconds taken Functions:
public makeVideo(): returns a Video generated by combining lastImages
3.2.3.
CloudComputation Package Classes
Class: ComputationManager
uses the AWS cloud API to send the captured image frames to the appropriate models on the cloud for processing, based on user preferences, and receiving the inference results from the cloud
Attributes:
public ModelManager modelManager Functions:
public receiveImage(): returns the taken Image taken by CameraController to classes using AWS API
public receiveFeatureSelection(): returns the users’ feature selections to classes using AWS API as ArrayList<String>
public loadNecessaryModels(ArrayList<String> selectedFeatures): updates models to be used by AWS and mobile phone according to priorities and capabilities
public sendComputationResults(Map<String str, Object obj): send results as a mapped model name and result pair to be displayed
3.2.4.
Models Package Classes
Class: ModelManager
responsible for keeping track of the active models, which varies based on user preference, transferring the captured image frames to these models for inference and delegating the received outputs to the ResponseManager for giving appropriate responses to the user based on model inference results
Attributes:
private Map<String,Model> activeDetectors Mapped list of all active models with their model name
Functions:
public ModelManager(): loads active models according to the saved feature selections when application started
public feedImageToModels(Image cameraImage): gets camera image and feeds it to the active models to get results
public activateModel(String modelName): when a new Feature Selected activates related models and add them to the activeDetectors
Class: Model
a wrapper for the Tensorflow's Classifier class and it is responsible for providing model inference.
Attributes:
private String modelName name of the model used as Classifier private Classifier model loaded Classifier by using file Path Functions:
public Model(String filePath):constructor of Model when called sets the modelName and loads Classifier according to filePath
public infer(Image img): infers the model on to the Image taken and returns the result of the Model as Object
public getModelName(): returns name of the used model as String
3.3. Data Tier Classe
3.3.1.
AWS Cloud Computing Server Package Classes
Class: CloudPredictor
This class manages the process of requesting the input of a model that is running on AWS cloud. The results of the models are requested by using HTTP requests. This class mainly sends the footage taken by the application and sends it to the cloud, then returns a
corresponding result to the application. This class serves as a driver function for interacting with the AWS Cloud Computing server.
Attributes:
private String modelPath is the path of the requested model which is the oth where the
model is located in the AWS cloud.
private AmazonSageMakerRuntime client is the client instance for operating with the
AWS server for sending HTTP requests to the server and taking corresponding results. Communication with the AWS server can only be done using this instance.
Functions:
public CloudPredictor(String path) is the constructor of the class which only takes the path of the model which is the location in the AWS server. This constructor initiates a client instance for connecting to AWS Cloud Computing server where models are able to run.
public getModelOutput(ArrayList<Image> record) is the method for retrieving an output from a model by using an input video as a series of images. The only input for this method is the series of images to be analyzed. The HTTP request is builded by using the path of the model file, which is an attribute of this class
3.3.2.
Google Drive Server Package Classes
Class: DriveManager
This class manages the operation of uploading video logs to the Google Drive account of the user by authenticating to the server.
Attributes:
private MediaHttpUploader uploader is the helper class provided by Google Drive API
which performs the upload operation to the Google Drive server.
private Credential userCredential is the object that holds the user credentials of the user
private File logFile is the file that contains the Video log to be saved to the cloud. The
video is converted to a File type for upload.
private DataStoreFactory dataStoreFactory is the class instance for accessing to
Google Drive storage. Provided by Google Drive API.
private String uploadDirectory is the directory in the Google Drive where the file is going
to be uploaded.
private HttpTransport httpTransport is the instance of the helper class that is going to be
used to establish HTTP connection with the Google Drive server.
private Drive driveClient is the client instance that is going to be used to establish the
connection with the Google Drive server.
Functions:
public DriveManager() is the constructor for saving the Google Drive credentials initially. public DriveManager(VideoLog log, String uploadPath) is the constructor which takes a VideoLog instance and a path to upload the video log to.
public void createAuthEntry(String username, String password) is the method for
creating a Google Drive credentials and saves these credentials inside the application.
private void authenticate() is the method for fetching the credentials information from the
local storage and initializing the credentials object accordingly.
private void createFile(VideoLog log) creates a file object from the VidoLog instance to be saved to Google Drive Server.
public void uploadLog() uploads the logFile to the Google Drive Server.
3.3.3.
VideoLog Package Classes
Class: VideoLog
This class represents the video logs to be saved either to the local storage or Google Drive Server.
Attributes:
private File directory is the object that represents the directory that the file is going to be
saved to.
private String fileName is the name of the videoLog that is going to be saved to the
storage unit.
private Intent videoData is the data object that stores the video data, which android
system uses it to convert to a video file.
Functions:
public VideoLog(File directory, Intent videoData) is the constructor of the class which generates a fileName for the file and sets the other attributes
private void setVideoContent(Intent data) is the setter method for setting the video data for this class
private String generateFileName() is the file name generator that uses the metadata from the trip to generate a log file name.
public String getFileName() is the getter method for accessing the name of the file.
Class: VideoLogManager
The driver class for managing the file saving operations for the VideoLog class instances
Attributes:
private DriveManager cloudManager is the instance of the Google Drive driver that is
used to manage the Google Drive operations.
private VideoLog log is the instance of the log that is going to be saved
private String preference is the attribute representing whether the user prefers to save
the file to the Google Drive server or willing to save the file locally
Functions:
public VideoLogManager(VideoLog log) is the constructor of the class which takes only the VideoLog to be saved, which represents the video to be saved
private void recordLocalLog() is the method for saving a video log to the local storage of the mobile device.
public void recordlLog() is the method for recording the video log either to the local storage or Google Drive storage.
3.3.4.
SelectedFeatures Package Classes
Class: SelectedFeatures
This class represents and the features selected by the user and provides easy access accordingly.
Attributes:
private ArrayList<ModelData> modelsSelected is the list of the models selected in the
form of a modelData class
Functions:
public SelectedFeatures(String featurePath) is the constructor that fetches the features
selected by the user from the loca storage and builds the modelData objects respectively.
public ModelData getModel(String modelName) is the getter method to retrieve a model with a specific name in the form of ModelData object.
This class manages the features selected by the user by modifying the file allocated to the selected features by the user.
Attributes:
public SelectedFeatures features is the list of features selected which can only be
modified by this class
Functions:
public FeatureSelectionManager(String featurePath) fetches the features selected by
the user from the local storage and constructs the object accordingly.
public updateSelection(Array<ModelData>) updates the selection of the user in terms of features used and serializes this selection.
Class: ModelData
This class represents all the data necessary to run the model, which are the name of the model and the path that the model is located.
Attributes:
private String modelName is the name of the model which is used to identify the model private String localModelPath is the path where the model file is stored in the local
storage
private String cloudModelPath is the path where the model is stored in the AWS Cloud
storage
Functions:
public ModelData(String modelName, String localPath, String cloudPath) is the
constructor of the ModelData object which initiates the attributes of the object.
public getModelPath(String computeDest) returns the respective path according to the preference whether the model is going to be run on the local device or AWS Clıud Computing server.
4.
Glossary
As the abbreviations are already defined in section 1.4, this section contains the information that in which pages these words are being used.
Machine Learning Model, 4,5,10,11,12,13,14,15,17,20,23,24,25,26,28,29 AWS, 3,4,5,10,14,15,24,25,29
Detection, 3,5,21
API, 5,10,11,13,15,21,24,26 TensorFlowLite, 5
5.
References
[1]
“Introduction to Object-Orientation and the UML,” Agile Data. [Online]. Available:
http://www.agiledata.org/essays/objectOrientation101.html. [Accessed: 17-Feb-2020].
[2]
“Citation Styles: APA, MLA, Chicago, Turabian, IEEE: IEEE Style,” LibGuides. [Online].
Available: https://pitt.libguides.com/citationhelp/ieee. [Accessed: 17-Feb-2020].
[3]
“Mediator,” Refactoring.Guru. [Online]. Available:
https://refactoring.guru/design-patterns/mediator. [Accessed: 17-Feb-2020].
[4]
