Laboratory on creation of large web‑services and high‑loaded solutions for the Internet

Services

Development of large web‑services
Creation of Hi‑Load projects
Development of mobile apps
System administration
Complex product testing
Site security audit

Our largest projects

Task:

Data import/export from Yandex.Market and other marketplaces

Result:

The system of import/export of goods has been developed and adjusted. It process tens of gigabytes of information per day without significant pressure on the server equipment.

Description:

— The ability to perform automatic imports according to a schedule specified by the client (daily, once a week, once a month) has been developed;
— automatic synchronization of information from the main site: discounts, price and availability changes, photo updates, etc.

Task:

Creation of CRM system

Result:

Our own, expanded CRM system was developed. It automates interaction with customers and allows managing internal business processes.

Description:

— manager's personal account provides full interaction with customer by next functions: automatic customer identification during an incoming call, outcoming calls, history of interaction with the customer recorded in notes, invoicing, access to statistics of client’s visits and orders;
— there is an archive of records of client’s dialogues in order to control the quality of managers’ work;
— attendance record has been developed;
— the ability of analysis of various types of statistics displayed in a detailed report was implemented.

Task:

Site constructor

Result:

Site constructor with flexible settings has been created.

Description:

— there are different variants of website design (more than 100 templates, the ability to set-up an individual header);
— advanced settings of online store give the possibility to place there more than 300 000 products, basket of buying and orders, and allows the usage of various payment and delivery terms;
— acceptance of electronic payment on the site, integration with credit aggregators.

Task:

Rubricator with the support of tens of thousands rubrics

Result:

A convenient and functional rubricator with a multi-level structure and flexible settings system has been created.

Description:

— the ability to add / remove the necessary rubrics, as well as to change their structure directly through the admin panel;
— opportunity of addition of parameters necessary to the customer to characteristics of goods;
— the possibility of creating temporary rubrics and sections for the promotion of seasonal products, goods of certain topics;
— creation of thematic pages with products for their additional promotion.

Target:

creation of the system monitoring the work of the equipment and apps

Features:

— extended metric detalization;
— high speed of data processing;
— modern and convenient web-interface;
— the ability to add your own metrics written in Python;
— notification by SMS, Telegram, Email, etc.

Description:

The following number of technologies was chosen for realisation of this project:

— Sensu – core monitoring system;
— InfluxDB – data base for time-series data storage;
— Grafana – web-interface;
— Chronograf – web-interface for direct data analysis in InfluxDB;
— Python – programming language for your own metrics;
— PHP – a programming language for writing an automated analysis system of state of apps and equipment, as well as incident alert system;

At present time the system assembles more than 250 unique parameters and processes about 20 000 metricss per minute.

Target:

creation of fault tolerant scalable file storage for storing user files.

Features:

— scalability;
— fault tolerance;
— high speed of operation.

Description:

A study of existing systems of creating DFS was conducted in order to implement this project.

Based on test results Ceph + CephFS was chosen, on the ground of which the storage on 1.8Tb SSD disks was developed.

Currently, the storage can hold 10Tb of data with replication factor 2.

The possibility of increasing the storage capacity till 40Tb within current equipment was provided.

Target:

to create a system for automatic scaling, optimizing, and watermarking images of the users of the site

Features:

— high quality of images;
— minimum weight of images;
— work with JPEG, PNG, GIF formats;
— high speed of operation;
— laying clients' individual watermarks;
— image caching.

Description:

It was decided to write our own programming in order to implement this project.

Programming language Go was chosen as the main one.

It helped to develop a fast HTTP server that immediately scales images to the desired size, lays a watermark and optimizes the size of the final file.

A simple distributed storage GlusterFS was implemented for the cache in order to organize fault tolerance.

At the moment, 3 similar servers process more than 1000 images per second without any problem, with 10 milliseconds as the average time for processing of one image.

Target:

to implement an incremental backup and restore system of MySQL DB.

Features:

— DB with over 200GB;
— table backup;
— basic daily backup;
— incremental backups every 30 minutes;
— restoration of required table at a specified time.

Description:

The innobackupex utility was taken as the basis for backup and recovery, a Python was used for writing a program of automatic table backup and a script for data recovery at a specified time.

Target:

to implement an automatic classification of goods according to categories

Features:

— high precision;
— text input data: product name, product folder name;
— high speed of operation.

Description:

This functionality was implemented using teaching the machine.

Data was collected from 3000000 products from 3000 categories, a primary analysis of the data was carried out, a list of stopwords that had not participated in the training was composed after the primary analysis.

Later, 80000 of best parameters for teaching the neural network were determined.

The accuracy of the classifier on the test dataset was 95%.

An HTTP server to access the API of this neural network was written within this task.

The average time for classification on a CPU is 50 milliseconds, and 3 milliseconds on an NVIDIA GTX 1080 GPU.