A Service-based Preset Recommendation System for Image Stylization Applications

A Service-based Preset Recommendation System
for Image Stylization Applications
Florian Fregien1, Fabian Galandi1, Max Reimann1, Sebastian Pasewaldt2,Jürgen Döllner1 and Matthias Trapp1
1Hasso Plattner Institute, Faculty of Digital Engineering, University of Potsdam, Germany
2Digital Masterpieces GmbH, Potsdam, Germany
7th International Conference on Human Computer Interaction Theory and Applications (HUCAPP 2023)

Image Stylization Applications
2/20/2023 A Service-based Preset Recommendation System for Image Stylization Applications 2
Browser-based application that stylize images using service-based GPU-processing deployed on AWS.

Image Stylization Applications
Browser-based applications that
stylize images using service-based
GPU-processing deployed on AWS.

Level-of-Control for Image Editing
Graphical User Interfaces for „Casual Creativity“ on Mobile Phones
Numerous parameters Parameters + presets Presets only
Tobias Isenberg. “Interactive NPAR: What type of tools should we create?”
In: Proceedings of the International Symposium on Non-Photorealistic
Animation and Rendering (NPAR as part of Expressive, May 7–9, Lisbon,
Portugal)., Eurographics Association, 2016, pp. 89–96

Problem Statement
State-of-the-art in image stylization apps:
 High number of available filter operations
 Each operation is highly-configurable ...
 increase operating complexity
 require time-consuming trial and error
 enable customization/individual styles
Long-term Objective and Use-Cases:
compute recommendations based on…
… most used filter operation configurations
… most used operations (individually)
… the input image characteristics
Exemplary results obtained
by an image stylization app.

Approach
Recommender
System
Visual Media Descriptors
Filter Operation Settings
User
Usage Data

Approach
App Database
1. Transmit usage data
3. Deliver recommendations
2. Analyze usage data

Classes of Recommendation Systems
likes
similar
recommended
Alternative approaches: Demographic Filtering, Knowledge-based, Hybrid approach
[Rocca, B., 2019] Goyani, M.M., & Chaurasiya, N., 2020]
Collaborative Filtering
Content-based Filtering
likes
similar
recommended
likes
likes

Recommendation System Challenges
Cold Start Problem Data Sparsity Scalability Adaptability

Assessment of DBMS Approaches
Relational DB:
➖ Limited structural flexibility due to DB schema
➖ Expensive JOIN operations on large tables
Document-based DB (NoSQL):
➕ Structural flexibility due to schema-less model
➖ Difficult to express and represent related data (many duplicates, hard to analyze)
Graph-based DB (NoSQL):
➕ Structural flexibility due to schema-less model
➕ No JOIN operation required (performant storage & querying of relations)

Database Modelling
IS_UPLOADED_BY
:User
id
IS_UPLOADED_FROM
:App
name
platform
version
:Medium
type: image
IS_DESCRIBED_BY
:Metadata
type: EXIF
fLength: 4.25
cSpace: 65535
:Metadata
type: JFIF
xDensity: 72
yDensity: 72
:Metadata
type: GPS
latitude: 52.382
longitude: 13.08
CONTAINS
order: 0
CONTAINS
order: 1
:Operation
id: oil_0
preset
:Operation
id: styleTransfer
preset
ADJUSTED_BY :Parameter
name: details
value: 8.5
:Parameter
name: contrast
value: 1
IS_PROCESSED_WITH
:Pipeline
preset:
winterMarcOil

Comparison of Graph-based DBMS
Aspect Neo4j Dgraph TigerGraph JanusGraph
Open Source ✅ ✅ ❌ ✅
Self hostable ✅ ✅ ❌ ✅
Stable / Mature Ecosystem ✅ ✅ ✅ ❌
API for JavaScript ✅ ✅ ❌ ❌
Complex Graph / ML algorithms ✅ ❌ ✅ ❌

Microservice Architecture Integration
Microservice API
API
Gateway
Auth Manager
Image Processor
API Documenter
API Keys DB
+
Usage Data Collector
Usage Data Analyzer
Application
(Mobile Web)

Workflow
Usage Data Analyzer
3. Request recommendations using descriptor
Usage Data Collector
1. Transmit descriptor & operation configuration 2. Store usage data
Application
(Mobile Web)y
4. Analyze

Web Application – User Interface
A: Image Canvas
 Image im-/export
 Image exploration
B: Operation View
 List of operations
 Operation selection
C: Option View
 Preset selection
 Operation settings
D: Pipeline View
 Combine operations
 Import/export
A
B
C
D

Web Application – Integration (I)

Web Application – Integration (II)

Web Application – Integration (III)

Performance – Dataset & Measurement
Experiment:
 Importing usage data into an empty database:
1. Measure the runtime of each import() query
2. Measure the runtime of querying the Top-10
operations among all applications
System Specification:
Dataset:
 11,609 entries ≈ 40 nodes per entry
 462,190 nodes in database:
 472 Application nodes
 11,594 Medium, Pipeline nodes
 35,696 Metadata nodes
 54,345 Operation nodes
 348,488 Parameter nodes
 Storage size: 75.26 MB
CPU Intel® Core™ i9-10900K @ 3.70 GHz
Cores 10 (20 logical)
RAM 64 GB
Runtime Node.js 18.6.0
Database Neo4j 4.4.7

Performance – Import Usage Data

Performance – Most Used Operations

Conclusions & Future Work
 Approach for service-based out-of-core preset recommendation system
 Further evaluation and user testing required
 Fundamental building block for more sophisticated recommender approaches
View publication stats

A Service-based Preset Recommendation System for Image Stylization Applications

More Related Content

Similar to A Service-based Preset Recommendation System for Image Stylization Applications (20)

More from Matthias Trapp (20)

Recently uploaded (20)

A Service-based Preset Recommendation System for Image Stylization Applications