Vaadin Flow enables the development of complete web applications exclusively in Java. Components, layouts, navigation, and even complex UI structures can be modelled on the server side without working directly with HTML or JavaScript. This approach is one of the main reasons why Vaadin is especially popular in Java-centric projects.

However, as a project grows in size, a typical problem arises: styling is often done directly in Java code. Vaadin allows this conveniently via the getStyle().set(…) method. This allows you to adjust spacing, colours, or layout properties quickly. In small prototypes, this procedure initially seems harmless and even practical.

1. The problem with inline styling in Vaadin Flow
2. Separation of structure and presentation: Java vs CSS
   1. Structure belongs in Java.
   2. Appearance belongs in CSS.
   3. Advantages of this separation
3. Replace inline styles with CSS classes
   1. Case in point
      1. Step 1: Introduce a CSS class
      2. Step 2: Using the class in Java code
   2. Another example from a view
   3. Benefits of this approach
4. Structure of CSS Files in a Vaadin Flow Application
   1. The frontend directory
   2. Integration of styles in Vaadin
   3. Separation of layout and component styles
   4. Consistent naming of CSS classes
5. Refactoring Real Views: Examples from the URL Shortener
   1. Example 1: MainLayout
   2. Example 2: Map layout in the AboutView
   3. Example 3: Structured layout classes
   4. Refactoring as an incremental process
6. Best Practices for CSS in Vaadin Flow Applications
   1. Java describes structure – CSS describes representation
   2. CSS Classes Instead of Inline Styles
   3. Leveraging Lumo Design Variables
   4. Semantic Class Names
   5. Use responsive layouts consciously
   6. Inline Styles as a Deliberately Used Exception
7. Conclusion of the refactoring

In larger applications, however, this practice quickly leads to a code state that is difficult to maintain. Styling information is spread over numerous views and components. Changes to the design have to be made throughout the Java code. Reusability of styles becomes virtually impossible, and the distinction between UI logic and presentation layer becomes increasingly blurred.

A typical example in many Vaadin projects looks something like this:

component.getStyle()   .set("padding","var(--lumo-space-m)")   .set("border-radius","var(--lumo-border-radius-l)");

Such constructs can often be found in almost every view. Over time, a mixture of layout logic, component structure and CSS definitions emerges directly in Java code.

In the open-source project URL-Shortener, exactly this problem became apparent. Several views contained inline styling, which was originally intended for quick UI adjustments. However, as functionality grew, it became clear that this pattern is not sustainable in the long term.

The goal of the following refactoring was therefore clearly defined:

• Remove inline styling from views
• Introduction of clear CSS classes
• Clean separation between structure (Java) and presentation (CSS)
• Better maintainability and reusability of the UI

The article describes step by step how an existing Vaadin UI can be converted from inline styling to a structured CSS architecture. The examples come directly from the real application and show typical situations that occur in many Vaadin projects.

This is not about making Vaadin completely CSS-driven. Vaadin remains a server-side UI framework. Rather, it is about establishing a clear division of responsibilities: Java defines the structure of the user interface, while CSS handles its visual design.

This seemingly small change has a significant impact on the maintainability of an application – especially if a Vaadin application is developed over the years.

The problem with inline styling in Vaadin Flow

At first glance, the getStyle().set(…) method in Vaadin Flow seems extremely practical. With just a few lines, spacing can be adjusted, layout problems corrected, or colours changed – directly in the Java code of the respective view. Especially in early project phases or during prototype development, this approach seems quick and uncomplicated.

In real applications, however, a different development often emerges. As a project grows in size, inline styles spread across multiple views. What were initially only small adjustments gradually became a large number of local layout and styling corrections that are implemented directly in the Java code.

For example, a typical snippet from a Vaadin application might look like this:

component.getStyle().set("padding","var(--lumo-space-m)");layout.getStyle().set("gap","var(--lumo-space-l)");button.getStyle().set("margin-left","auto");

Each of these lines is harmless in itself. Taken together, however, they lead to several structural problems.

Firstly, the styling is spread over the entire application. Instead of a central place where the visual appearance is defined, design decisions are spread across many different Java classes. If, for example, the spacing of a layout or the display of certain components is to change later, developers must first find the appropriate places in the code.

Secondly, reusability is made more difficult. Many layout patterns – such as map layouts, typical spacing, or alignments – appear multiple times in an application. However, if these are defined as inline styles, they exist again in each view. Changes then have to be made in several places.

Third, inline styling blurs the architectural separation between structure and representation. Java classes actually define the structure of the user interface, orchestrate components, and respond to user interactions. When CSS properties are integrated directly into these classes, presentation logic mixes with the UI’s structural definition.

This problem is particularly evident in larger Vaadin applications developed over a longer period. Small visual adjustments accumulate, and individual getStyle().set(…) calls, an implicit styling strategy gradually emerges – spread across numerous classes.

In the open-source project URL-Shortener, exactly this pattern emerged. In several views, including MainLayout, AboutView and CreateView, layout and styling adjustments were made directly in Java code. Although the application remained functionally correct, the UI’s visual behaviour became increasingly difficult to understand.

The refactoring was therefore not just aimed at removing individual style calls. Instead, a clear rule should be introduced:

Java defines the structure of the user interface – CSS defines its visual design.

At first, this separation seems like a small organisational change. In practice, however, it significantly improves the application’s maintainability. Styles can be defined centrally, reused, and used consistently across multiple views.

The following sections show step by step how this separation can be implemented in an existing Vaadin Flow application.

Separation of structure and presentation: Java vs CSS

Now that the problems of inline styling in Vaadin Flow have become clear, the central question arises: How can a clean separation be created between the structure of the user interface and its visual representation?

In classic web applications, this division is clearly defined. HTML describes the structure of the page, CSS controls the visual appearance, and JavaScript handles interactions and logic. In Vaadin Flow, this architecture shifts slightly because the UI structure is defined entirely in Java. Nevertheless, the basic idea remains: structure and presentation should be kept separate.

Structure belongs in Java.

Java defines the structure of the user interface in a Vaadin application. These include, in particular:

the composition of components

• Layout Structures
• Navigation and routing
• Event Handling
• Data Binding

A typical example is the structure of a layout:

VerticalLayoutcontainer=newVerticalLayout();container.add(newH2("Create new short links"),form,actions);

Here, the code describes only the structure of the interface: which components are present and how they are arranged.

Appearance belongs in CSS.

The visual representation of the components, on the other hand, should be defined via CSS. These include, but are not limited to:

• Spacing and padding
• Colours and backgrounds
• Shadows and Borders
• Typography
• responsive layout adjustments

For example, instead of setting padding directly in Java code

component.getStyle().set(“padding”, “var(–lumo-space-m)”);

A CSS class is used:

component.addClassName(“card”);

The actual visual definition is then in the stylesheet:

.card{ padding:var(--lumo-space-m); border-radius:var(--lumo-border-radius-l);}

This division ensures that layout decisions can be managed centrally rather than spread across several views.

Advantages of this separation

The consistent separation between Java structure and CSS presentation has several advantages.

First, the Java code will be much more readable. Views focus on the composition of the UI components without being overloaded by styling details.

Second, styles can be centrally managed and reused. Once a CSS class has been defined, it can be used across different views without layout rules repeating in the code.

Third, the application’s maintainability is improved. Changes to the design do not require adjustments in numerous Java classes, but can be made centrally in the stylesheet.

Especially in larger Vaadin projects with several developers, it quickly becomes apparent how important this clear division of responsibility is.

In the next section, we will use concrete examples from the application to show how existing inline styles can be replaced by CSS classes step by step.

Replace inline styles with CSS classes

Once the basic separation between structure (Java) and presentation (CSS) has been defined, a practical question arises in existing Vaadin applications: How can an existing codebase be gradually converted from inline styling to CSS classes?

In practice, the changeover usually takes place incrementally. Instead of refactoring the entire application at once, individual views or components are cleaned up first. Direct style definitions in the Java code are removed and replaced by CSS classes.

Case in point

A common pattern in Vaadin views is to control layout behaviour directly via inline styles. An example from the application is moving a component to the right end of a layout.

For example, before refactoring, the code looked like this:

storeIndicator.getStyle().set(“margin-left”, “auto”);

The code serves its purpose: the component is moved to the right. The problem, however, is that this layout rule is now firmly anchored in Java code.

Step 1: Introduce a CSS class

Instead of setting the Style property directly, a suitable CSS class is first defined. This is typically located in the application’s global stylesheet file.

.mainlayout-right{ margin-left:auto;}

The advantage of this solution is that the layout rule is now centrally defined and can be reused by several components.

Step 2: Using the class in Java code

The next step is to remove the inline style in the Java code and replace it with the CSS class.

storeIndicator.addClassName(“mainlayout-right”);

This means that the Java code remains solely responsible for the UI’s structure. The component’s visual alignment is controlled entirely by CSS.

Another example from a view

Larger layout definitions can also be cleaned up in this way. In many Vaadin views, for example, constructs such as:

container.getStyle()       .set("border-radius","var(--lumo-border-radius-l)")       .set("gap","var(--lumo-space-l)");

These style definitions can be converted into a CSS class:

.card-container{ border-radius:var(--lumo-border-radius-l); gap:var(--lumo-space-l);}

In Java code, the definition is then reduced to:

container.addClassName(“card-container”);

Benefits of this approach

This refactoring results in several improvements.

First , the Java code will be much more compact and easier to read. Layout details disappear from the View classes.

Second , styles can be managed centrally. Layout changes only need to be made at one point in the stylesheet.

Third , it creates a consistent visual language within the application by leveraging reusable CSS classes.

The migration of inline styles to CSS classes is therefore a central step in keeping Vaadin applications maintainable in the long term.

The next chapter shows where CSS files are stored in a Vaadin Flow application and how they are correctly integrated.

Structure of CSS Files in a Vaadin Flow Application

Now that inline styles have been removed from the Java code and replaced with CSS classes, the next important question arises: Where should these CSS definitions be stored in a Vaadin flow application?

Vaadin Flow is internally based on Web Components and uses the application’s frontend directory to provide static resources such as stylesheets, JavaScript, or images. For CSS, this results in a clear structure that has proven itself in practice.

The frontend directory

In a typical Vaadin application, there is a directory called frontend. This contains all resources that are loaded directly by the browser.

For example, a commonly used structure looks like this:

frontend/

** styles/**

** main.css**

** layout.css**

** components.css**

Within this folder, the styles can be logically organised by area of responsibility.

• main.css contains basic styles of the application
• layout.css defines layout rules
• components.css includes reusable component styles

This splitting prevents all styles from ending up in a single file and facilitates long-term maintenance.

Integration of styles in Vaadin

For the CSS files to take effect in the application, they must be loaded by Vaadin. This is typically done via annotations such as @CssImport.

An example in a central layout class might look like this:

@CssImport("./styles/main.css")publicclassMainLayoutextendsAppLayout{   ...}

Once this annotation is in place, the styles defined in it become available throughout the application.

Alternatively, you can use a global theme, in which styles are loaded automatically. In this case, the CSS files are located in the directory, for example:

frontend/themes//

This approach is particularly recommended in larger applications, as the theme serves as a central point for all design decisions.

Separation of layout and component styles

Another best practice is to distinguish between layout styles and component styles.

Layout styles describe, for example:

• Container spacing
• Grid or Flex Layouts
• responsive behavior

Component styles, on the other hand, define the visual appearance of reusable UI elements such as cards, badges, or toolbars.

This separation prevents layout rules and visual component logic from being mixed.

Consistent naming of CSS classes

To ensure that CSS classes remain understandable in the long term, a consistent naming convention should be used.

One possible pattern is based on the respective view or component:

.about-card

.about-hero

.mainlayout-right

.searchbar-container

This makes it immediately recognisable which area of the application a certain style belongs to.

A clearly structured CSS organisation is the basis for ensuring that the previously introduced separation between Java and CSS is permanent.

While Java continues to define the structure and behaviour of the user interface, the application’s visual appearance is centrally controlled via stylesheets. This reduces redundancies, facilitates design changes, and ensures a consistent user interface across all views.

The next chapter shows how concrete refactorings in the application were implemented and which patterns proved particularly helpful.

Refactoring Real Views: Examples from the URL Shortener

The principles described so far – the separation of structure and presentation, as well as the replacement of inline styles with CSS classes – can best be understood through concrete examples. In the open-source project URL-Shortener, several views were gradually refactored to implement this separation.

This chapter shows how existing Vaadin code has been adapted and which patterns have proven to be particularly practical.

Example 1: MainLayout

In the MainLayout, an inline style was originally used to move an element to the right within a layout.

Before refactoring, the code looked like this:

storeIndicator.getStyle().set(“margin-left”, “auto”);

This solution works technically flawlessly, but leads to layout rules being defined directly in Java code.

After the refactoring, a CSS class was introduced instead.

Java:

storeIndicator.addClassName(“layout-spacer-right”);

CSS:

.layout-spacer-right{ margin-left:auto;}

The layout rule is now exclusively in the stylesheet, while the Java code only contains the structural information that this component has a specific layout role.

Example 2: Map layout in the AboutView

The AboutView uses multiple containers that are visually represented as maps. Originally, these layout rules were defined directly via style calls.

Typical inline code looked something like this:

card.getStyle()   .set("background","var(--lumo-base-color)")   .set("border-radius","var(--lumo-border-radius-l)")   .set("box-shadow","var(--lumo-box-shadow-s)");

These styles were then converted into a reusable CSS class.

CSS:

.card{ background:var(--lumo-base-color); border-radius:var(--lumo-border-radius-l); box-shadow:var(--lumo-box-shadow-s); padding:var(--lumo-space-l);}

Java:

card.addClassName(“card”);

The advantage is that all cards in the application now have a consistent appearance, and changes can be made centrally.

Example 3: Structured layout classes

Another pattern is to name layout roles via CSS classes explicitly. Instead of generic styles, semantic classes are introduced to describe an element’s purpose.

Examples include:

.searchbar-container

.bulk-actions-bar

.overview-grid

Such classes make it easier to understand the application’s CSS layout.

Refactoring as an incremental process

It is important to note that this change need not occur in a single step. In practice, an incremental approach has proven to be effective.

• New components are developed directly with CSS classes
• Existing inline styles will be refactored when the opportunity arises
• Recurring style patterns are gradually centralised

In this way, an existing Vaadin application can be converted into a cleanly structured CSS architecture without major risks.

The examples from the URL shortener show that even small refactorings can have a significant impact on an application’s maintainability. Removing inline styles makes the Java code clearer, while the visual design can be maintained centrally in the stylesheet.

The next chapter identifies the general best practices that can be derived from this refactoring and how to apply them in future Vaadin projects.

Best Practices for CSS in Vaadin Flow Applications

After the refactoring of the existing views is complete, the question arises: which general rules can be derived from this? In practice, it has been shown that a few simple principles are already sufficient to keep Vaadin Flow applications maintainable and consistent in the long term.

This chapter summarises the most important best practices that have proven themselves during the conversion of the URL Shortener project.

Java describes structure – CSS describes representation

The most important rule is also the simplest: Java defines the structure of the user interface, CSS defines its visual design.

In Vaadin, the entire UI composition is modelled in Java. Layout containers, components and their hierarchy therefore clearly belong in the Java code.

Examples of structural definitions in Java include:

• Layout containers (VerticalLayout, HorizontalLayout, FormLayout)
• Component Structure
• Event Handling
• Data Binding

Visual aspects, on the other hand, should be regulated via CSS. These include, in particular:

• Spacing
• Colors
• Shadows
• Typography
• visual highlights

This clear separation prevents UI logic and presentation details from being mixed in the same code.

CSS Classes Instead of Inline Styles

Inline styles should only be used in exceptional cases. Instead, it is recommended to define consistent CSS classes and bind them to components via addClassName().

Example:

component.addClassName(“card”);

The visual definition is then done in the stylesheet:

.card{ padding:var(--lumo-space-m); border-radius:var(--lumo-border-radius-l);}

This approach ensures that layout rules can be maintained centrally.

Leveraging Lumo Design Variables

Vaadin uses Lumo, a consistent design system that provides numerous CSS variables.

Typical examples are:

• --lumo-space-m
• --lumo-border-radius-l
• --lumo-box-shadow-s
• --lumo-primary-color

These variables should be used preferably, as they automatically harmonise with the chosen theme and ensure consistent spacing and colours.

An example:

.card{ padding:var(--lumo-space-l); border-radius:var(--lumo-border-radius-l);}

Semantic Class Names

CSS classes should not only describe how something looks, but what role an element plays in the layout.

Less helpful would be, for example, names like:

• .box1
• .red-border
• .container-large

Semantic names that are based on views or components are better:

• .about-card
• .searchbar-container
• .bulk-actions-bar
• .overview-grid

This means that even in larger style sheets, it remains clear to which area of the application a style belongs.

Use responsive layouts consciously

Vaadin already provides many mechanisms for responsive layouts, including FormLayout, FlexLayout, and SplitLayout.

For example, FormLayouts can be configured directly via Java:

form.setResponsiveSteps(   newFormLayout.ResponsiveStep("0",1),   newFormLayout.ResponsiveStep("900px",2));

These structural layout decisions belong in the Java code by design, as they are part of the UI composition. CSS then only complements the visual behaviour.

Inline Styles as a Deliberately Used Exception

Despite all the recommendations, there are situations in which inline styles can be useful.

Typical examples are:

• dynamically calculated layout values
• short-term UI adjustments
• experimental prototypes

However, it is important that these cases remain the exception and do not become the default style of an application.

The combination of a clear division of responsibility, reusable CSS classes and consistent design variables results in a much more maintainable Vaadin application.

Especially in projects developed over a longer period, this structure quickly pays off. Changes to the visual design can be implemented centrally, while the Java code continues to describe only the user interface’s structure and behaviour.

Conclusion of the refactoring

The switch from inline styling to a clearly structured CSS architecture may seem like a small cosmetic change at first glance. In practice, however, it quickly becomes apparent that this refactoring has a profound impact on the maintainability and comprehensibility of a Vaadin Flow application.

In the original state of the application, many layout and display details were anchored directly in the Java code of the views. Method calls such as getStyle().set(…) provided quick UI adjustments in the short term but, in the long term, led to a mixing of structure and presentation. The result was a codebase in which design decisions were spread across numerous classes.

With the introduction of CSS classes and the consistent outsourcing of visual properties to style sheets, this mixing has been broken up again. Java now only describes the structure of the user interface: components, layout containers, navigation and interactions. The visual design, on the other hand, is defined centrally via CSS.

This separation brings several immediate benefits.

First, the Java code’s readability improves significantly. Views focus on the user interface’s composition without being overwhelmed by layout details. Developers can more quickly understand which structure a view has and which components interact with each other.

Second, the application’s design becomes more consistent. Reusable CSS classes ensure that similar components in different views are also visually identical. Changes to the appearance can be made centrally in the stylesheet without adapting numerous Java classes.

Thirdly, this structure facilitates team collaboration. Developers can focus on the Java side of the application, while design adjustments can be made independently in CSS. Especially in larger projects, this reduces conflicts and simplifies further UI development.

The refactoring of the URL shortener project has shown that even relatively small changes to the style structure can have a big impact on long-term maintainability. Especially in applications developed over the years, a clear separation between structure and presentation quickly pays off.

Vaadin Flow remains a server-side UI framework that puts Java at the centre of UI development. Nevertheless, such an architecture also benefits significantly from the classic principles of web development: structure, presentation, and behaviour should be clearly separated.

If you apply this basic rule consistently, you will get a Vaadin application that is not only functionally stable but also remains understandable and maintainable in the long term.

The starting point for this article was not a strategic architecture workshop or a long-term planned migration path, but a comparatively unspectacular step: updating the version numbers in the existing project. As part of further developing my URL shortener project, a regular dependency update was due anyway. Vaadin 25 (was already available at that time, so it made sense to proceed.

The source code for the project can be found and is

available underhttps://3g3.eu/url

The expectations were realistic, perhaps even slightly sceptical. A new major release, new minimum requirements, a modernised stack – all of these are usually good reasons to expect at least minor adjustments or initial friction. Accordingly, there was little hope that simply checking the version numbers would be sufficient.

1. Why Vaadin 25 is more than a version leap
2. The new minimum standard: Java 21, Jakarta EE and modern platforms
3. Leaner stack, faster builds and less ballast.
4. Styling & Theming Rethought: CSS Instead of Special Logic
   1. Before and after: styling in practice
      1. Example 1: Button styling
      2. Example 2: Layout Spacing and Consistency
      3. Example 3: Dark/Light theme without special logic
5. Component and rendering improvements in everyday life
   1. Before and After: Rendering and Overlay Behaviour in Practice
      1. Example 1: Nested dialogues
      2. Example 2: Focus return after dialogue closure
      3. Example 3: Overlay over Grid and Scroll Container
6. Element API, SVG and MathML: Technical UIs directly from Java
   1. Before and After: Technical Visualisation with and without Element API
      1. Example 1: Status indicator as SVG
      2. Example 2: Simple Bar Chart
      3. Example 3: Representation of a Formula with MathML

However, that was exactly the case. After adapting the Vaadin version and the current Java and platform dependencies, the project could be built, started, and operated without further changes. The application behaved as before: views were rendered correctly, dialogues worked, styles were intact, and even more complex UI flows showed no abnormalities.

This result was remarkable in that it was not based on a super trivial demo project. The URL shortener is a mature application with a clear module structure, server-side UI logic, security mechanisms and a small number of UI components. The fact that a major upgrade was possible under these conditions without immediate follow-up work is not routine.

It was precisely this experience that triggered a closer look at Vaadin 25. If a framework upgrade of this magnitude works right away, the question inevitably arises: why? What decisions in the substructure make this possible? Where was stability deliberately focused? And where are the changes lurking that only become relevant on closer inspection?

Why Vaadin 25 is more than a version leap

Vaadin 25 does not mark a classic evolutionary step with a collection of new widgets or API extensions, but a deliberate realignment of the framework. While previous major releases often featured additional features or incremental improvements, Vaadin 25 focuses on consolidation: fewer special logic elements, less legacy, and closer alignment with established standards in modern Java and web development.

This realignment is a direct response to the reality of many productive enterprise applications. Today, long-running web UIs must not only be functional but also maintainable, secure, and easy to integrate for years. This is exactly where Vaadin 25 comes in. The framework views itself less as an isolated ecosystem with its own rules and more as an integral part of a contemporary Java stack.

A central signal for this is the clear commitment to modern platforms. With Java 21 as the minimum requirement and the focus on current Jakarta and Spring versions, Vaadin is deliberately positioning itself where the rest of the enterprise Java stack is moving. Older compatibility promises are abandoned in favour of creating a clean, consistent technological underpinning.

Vaadin 25 is also a turning point conceptually. Many mechanisms that were once considered Vaadin-typical are now being questioned or simplified. Styling follows more classic CSS principles, build processes are approaching the standard Java workflow, and client-side code is becoming leaner and more transparent. The result is a framework that requires less explanation and integrates more seamlessly into existing development and operational processes.

Vaadin 25 is therefore not a release that primarily stands out for visible UI innovations. Its added value lies in its long-term perspective: more stable foundations, reduced complexity, and better integration with modern Java architectures. For developers and architects, this means one thing above all: In the future, decisions for or against Vaadin can be made more clearly along technical criteria – and less along framework specifics.

The new minimum standard: Java 21, Jakarta EE and modern platforms

With Vaadin 25, the framework draws a clear line and defines a new minimum standard for technology. This decision was made deliberately and is aimed at projects to be operated in the long term, using current Java platforms. Backward compatibility at all costs is abandoned in favour of clarity, consistency, and maintainability.

The focus is on Java 21 as a binding basis. Vaadin thus relies on the current LTS version, which brings both linguistic and runtime improvements. For Vaadin applications, this means not only access to modern language constructs but also a common denominator across UI code, backend logic, and tests. The days when UI code remained at an older language level for compatibility are over.

Closely linked to this is the alignment with current Jakarta standards. Vaadin 25 is based on Jakarta EE 11 and Servlet 6.1, and finally says goodbye to historical javax dependencies. At first glance, this change may seem like a purely technical detail. Still, in practice, it has noticeable effects: libraries, containers, and security mechanisms now share a consistent namespace and can be integrated more cleanly. Especially in safety-relevant applications, this reduces friction losses and misconfigurations.

This line will also be consistently continued in the Spring environment. Vaadin 25 is geared toward Spring Boot 4 and Spring Framework 7 and does not support earlier versions. This eliminates a significant amount of compatibility code required in previous versions. At the same time, Vaadin applications are moving closer to the mainstream Spring stack, which simplifies operation, monitoring and security integration.

The new minimum standard also affects the entire build and toolchain area. Modern Maven and Gradle plugins, current Node versions for the frontend, and a clearer separation between development and production artefacts make Vaadin projects feel more like classic Java applications. Special profiles and project-specific workarounds are becoming less important.

The perspective is important: Vaadin 25 requires more discipline at the start but reduces complexity over the long term. Projects planning to migrate to Java 21 or that have already migrated to Java 21 and current platforms will benefit directly. For older applications, on the other hand, the upgrade is a deliberate strategic decision – with initial effort but a much more stable foundation for the coming years.

Leaner stack, faster builds and less ballast.

One of the most noticeable effects of Vaadin 25 is not immediately visible in the UI; it is part of the project’s underlying architecture. The framework has been specifically streamlined in this version: transitive dependencies have been reduced, historical compatibility layers have been removed, and build processes have been simplified. The result is a stack that is much closer to a classic Java server application.

In previous Vaadin versions, it was not uncommon for projects to carry a significant amount of indirect dependencies. Many of these were necessary to support different platform versions, old browser strategies or alternative operating modes. With Vaadin 25, much of this ballast is eliminated. The dependencies are more clearly structured, more transparently comprehensible and overall more manageable.

This reduction directly affects the build process. Both Maven and Gradle builds benefit from shorter resolution times and less special configuration. In particular, the gap between development and production build has narrowed. Vaadin applications behave much more strongly than ordinary Java artefacts, making it easier for new developers to get started and simplifying CI pipelines.

Startup times in development mode also benefit from this approach. Fewer initialisation steps, clearer separation between server and client components, and modernised front-end dependencies result in a faster feedback loop. Especially in larger projects, where frequent restarts are part of the daily routine, this productivity gain should not be underestimated.

Another aspect is resource consumption during operations. A leaner stack does not automatically mean minimal memory usage, but it does reduce unnecessary load. Fewer libraries mean fewer classes, fewer potential conflicts, and a smaller attack surface. For production environments – especially in safety-critical or highly regulated contexts – this is a clear advantage.

The key thing is the demarcation: Vaadin 25 is not a minimal framework. Rather, it is about conscious reduction. Everything that is no longer necessary will be removed or simplified. This attitude runs through the entire stack and is a direct continuation of the repositioning described in the previous chapters.

The leaner stack thus forms an essential basis for further innovations in Vaadin 25. Simplified styling, more stable UI mechanisms and clearer migration paths are only possible because the framework frees itself from legacy burdens that have grown over the years.

Styling & Theming Rethought: CSS Instead of Special Logic

The styling model is one of the areas in which Vaadin 25 stands out most clearly from previous versions. While Vaadin has long brought its own concepts and abstractions around themes, variants, and style hooks, version 25 follows a clear course: CSS is understood as an equal, direct design mechanism – without unnecessary framework-specific paths.

This decision is less cosmetic than architectural. In many projects, the previous styling model required in-depth knowledge of Vaadin to make UI adjustments. At the same time, existing web know-how could only be used to a limited extent. Vaadin 25 specifically reduces this friction by aligning styling more closely with established web standards, making it easier to understand and maintain.

A visible result of this approach is the new default theme Aura. It does not replace Lumo, but sets a different focus. Aura looks calmer, more modern, and less playful without pushing itself to the forefront. For productive applications, this means a consistent look and feel without extensive customisation. Lumo remains available and is particularly suitable for projects that already build heavily on it or deliberately pursue a different visual profile.

In addition to the visual basis, flexibility is crucial. Vaadin 25 makes it possible to change themes dynamically – depending on user roles, clients or environments, for example. Light and dark variants can be realised without far-reaching modifications. This capability is not new, but it can be implemented more cleanly and comprehensibly with the simplified styling model.

The material theme was deliberately removed. This decision underscores the new focus: Instead of maintaining several design languages in parallel, Vaadin focuses on a few well-integrated core themes. For projects that require a highly individualised interface, this is not a disadvantage. On the contrary, a stronger CSS focus allows your design systems to be implemented more consistently and more framework-agnostic.

The effect on team collaboration is also important. Designers and front-end developers can work much more directly with Vaadin 25 without having to familiarise themselves with Vaadin-specific styling concepts. At the same time, Java developers retain full control over the UI’s structure and behaviour.

Before and after: styling in practice

To make the difference tangible, it is worth taking a direct look at typical styling approaches before and after Vaadin 25. The examples are deliberately simplified but clearly demonstrate the paradigm shift.

Example 1: Button styling

Before (classic Vaadin approach with theme variants and Java hooks):

javaCopy

Buttonsave=newButton("Save");save.addThemeVariants(ButtonVariant.LUMO_PRIMARY);save.getStyle().set("background-color","var(--lumo-success-color)");save.getStyle().set("color","white");

The styling here is partly anchored in the Java code. Colours and visual details are controlled via Vaadin-specific variables and APIs, which blur the separation between structure and representation.

After (Vaadin 25, CSS-first):

javaCopy

Buttonsave=newButton("Save");save.addClassName("btn-save");.btn-save{ background-color:var(--vaadin-color-success); colour:white;}

Responsibility is clearly separated: Java describes the structure and semantics, while CSS handles the visual appearance. The styling is easy to find, testable and customizable independently of the Java code.

Example 2: Layout Spacing and Consistency

Before (inline styles in code):

javaCopy

VerticalLayoutlayout=newVerticalLayout();layout.getStyle().set("padding","var(--lumo-space-m)");layout.getStyle().set("gap","var(--lumo-space-s)");

Such constructs are functional, but quickly lead to scattered styling knowledge in the code.

After (CSS-based layout classes):

javaCopy

VerticalLayoutlayout=newVerticalLayout();layout.addClassName("content-layout");.content-layout{ padding:1rem; Gap:0.5rem;}

Spacing and layout rules are now centrally defined and can be adapted consistently throughout the project.

Example 3: Dark/Light theme without special logic

Before (Vaadin-specific theme switch):

javaCopy

UI.getCurrent().getElement().setAttribute("theme","dark");

After (Vaadin 25, CSS-oriented):

javaCopy

UI.getCurrent().getElement().getClassList().add("theme-dark");.theme-dark{ --vaadin-color-background:#1e1e1e; --vaadin-color-text:#f5f5f5;}

Changing the theme is now a clearly defined CSS mechanism that can be easily integrated with existing design systems or user preferences.

These examples illustrate the core of the change: Vaadin 25 consistently shifts styling decisions to their proper place. The code becomes clearer, the styling more flexible, and collaboration among backend, frontend, and design roles much more relaxed.

Component and rendering improvements in everyday life

While many changes in Vaadin 25 are structural, the improvements to components and rendering are concrete in day-to-day UI work. In particular, dialogues, overlays, menus, and other floating UI elements benefit from a revised architecture designed for consistency, predictability, and stability.

In previous Vaadin versions, problems with overlaps, loss of focus or unexpected Z-index behaviour were not uncommon – especially in more complex applications with nested dialogues or dynamically reloaded components. Such effects could usually be remedied, but required additional logic, workarounds or a deep understanding of internal Vaadin mechanisms.

Vaadin 25 relies on a unified overlay model here. Dialogues, context menus, tooltips and similar components now follow a common rendering strategy. As a result, they are more consistently positioned, respond more stably to size changes, and behave more predictably when interacting with one another. For developers, this means fewer surprises and less special treatment in the code.

A particularly relevant aspect is focus management. Keyboard navigation, focus restoration after closing a dialogue, and simultaneous interaction with multiple overlays are much more robust in Vaadin 25. This not only improves the user experience but also enhances accessibility and testability.

The interaction with modern layouts also benefits from the new rendering logic. Overlays adapt better to viewport changes, respond cleanly to scroll containers, and retain their position even during dynamic UI updates. Especially in data-driven applications with many interactions, this reduces visual artefacts and inconsistencies.

It is important to note that these improvements hardly require any new APIs. Existing code often benefits from the upgrade itself. At the same time, previously necessary protective measures or auxiliary constructs become superfluous, simplifying the code and making it easier to read.

Before and After: Rendering and Overlay Behaviour in Practice

The differences between previous Vaadin versions and Vaadin 25 are particularly evident when examining typical UI scenarios in real-world applications.

Example 1: Nested dialogues

Before (classic approach with potential overlay issues):

javaCopy

DialogeditDialog=newDialog();editDialog.add(newTextField("Name"));DialogconfirmDialog=newDialog();confirmDialog.add(newSpan("Save changes?"));Buttonsave=newButton("Save",e->confirmDialog.open());editDialog.add(save);editDialog.open();

In more complex layouts, the second dialogue may not be positioned correctly on top of the first, or focus and keyboard navigation may be lost.

After (Vaadin 25, consistent overlay model):

javaCopy

DialogeditDialog=newDialog();editDialog.setModal(true);DialogconfirmDialog=newDialog();confirmDialog.setModal(true);Buttonsave=newButton("Save",e->confirmDialog.open());editDialog.add(save);editDialog.open();

Unified overlay rendering reliably stacks dialogues, passes focus correctly, and restores it after closing—without the need for additional auxiliary constructs.

Example 2: Focus return after dialogue closure

Before (manual focus correction necessary):

javaCopy

ButtonopenDialog=newButton("Edit");Dialogdialog=newDialog();dialog.addDialogCloseActionListener(e->openDialog.focus());openDialog.addClickListener(e->dialog.open());

Such patterns were necessary to ensure clean keyboard navigation.

After (Vaadin 25, automatic focus management):

javaCopy

ButtonopenDialog=newButton("Edit");Dialogdialog=newDialog();openDialog.addClickListener(e->dialog.open());

Vaadin 25 reliably handles focus return. The code becomes shorter, more understandable and less error-prone.

Example 3: Overlay over Grid and Scroll Container

Before (unexpected positioning for scroll containers):

xmlCopy

Grid<Item> grid = new Grid<>(Item.class);ContextMenu menu = new ContextMenu(grid);menu.addItem("Details", e -> showDetails());

Depending on the layout and scrolling behaviour, the context menu could appear offset or cut off.

After (Vaadin 25, more stable positioning):

xmlCopy

Grid<Item> grid = new Grid<>(Item.class);ContextMenu menu = new ContextMenu(grid);menu.setOpenOnClick(true);menu.addItem("Details", e -> showDetails());

The new rendering model handles scroll containers and viewport changes more consistently, so overlays appear where the user expects them.

These examples show that Vaadin 25 solves many UI problems not through new APIs, but through a more robust internal architecture. For existing applications, this often means less code, fewer workarounds, and a UI that behaves stably even in borderline cases.

Element API, SVG and MathML: Technical UIs directly from Java

With Vaadin 25, an area is gaining importance that has so far played only a minor role in many projects: direct work with the Element API. Native support for SVG and MathML significantly expands the scope for design – especially for technical, data-driven user interfaces where classic UI components reach their limits.

Until now, such requirements have often been the point at which additional JavaScript libraries or external front-end frameworks have been introduced. Diagrams, status graphics or mathematical representations could be integrated, but led to media breaks in the code and increased integration effort. Vaadin 25 noticeably reduces this need by exposing these technologies as full-fledged elements in the server-side UI model.

The Element API enables the creation and manipulation of structured DOM elements directly from Java. With the addition of SVG, graphical primitives such as lines, circles or paths can now also be modelled on the server side. For example, it can be used to create simple diagrams, status indicators or visual markers without leaving the Vaadin component model.

A similar effect is observed with MathML. Technical applications that have to represent formulas or calculation results benefit from the fact that mathematical expressions can be described semantically correctly. The display is not only visually more precise, but also more accessible – for example, for screen readers or automated tests.

The decisive point is not so much the technical feasibility as the architectural consistency. By integrating SVG and MathML into the Element API, the Java backend retains control over UI logic, rendering, and data flow. There is no additional client-side state that needs to be synchronised or secured. Especially in safety-critical or highly regulated environments, this advantage should not be underestimated.

Of course, this approach does not replace specialised charting libraries or complex visualisation tools. Vaadin 25 deliberately positions the Element API as a tool for targeted, controlled visualisations. When it comes to clear states, technical information or explanatory graphics, this approach is often more robust and maintainable than an external dependency.

Chapter 6 thus shows another facet of Vaadin 25’s realignment. The framework does not expand its capabilities through additional abstractions; instead, it provides direct access to established web standards. For developers, this means: more expressiveness, less integration effort and a UI that can be cleanly modelled beyond classic form and table views.

Before and After: Technical Visualisation with and without Element API

The advantages of the extended Element API are best understood through concrete examples. The following scenarios are typical of technical applications and illustrate the difference between traditional integration approaches and the Vaadin 25 solution.

Example 1: Status indicator as SVG

Before (external JavaScript library or client-side snippet):

xmlCopy

Div status = new Div();status.getElement().setProperty("innerHTML",  "<svgwidth='20'height='20'><circlecx='10'cy='10'r='8'fill='green'/></svg>");

The SVG code is embedded here as a string. Structure, semantics, and type checking are lost, and changes are error-prone.

After (Vaadin 25, SVG via Element API):

javaCopy

Elementsvg=newElement("svg");svg.setAttribute("width","20");svg.setAttribute("height","20");Elementcircle=newElement("circle");circle.setAttribute("cx","10");circle.setAttribute("cy","10");circle.setAttribute("r","8");circle.setAttribute("fill","green");svg.appendChild(circle);getElement().appendChild(svg);

The SVG is now a fully modelled DOM element. Changes can be made in a targeted manner, and the structure remains comprehensible and testable.

Example 2: Simple Bar Chart

Before (passing data to client-side code):

javaCopy

JsonObjectdata=Json.createObject();data.put("value",75);ui.getPage().executeJs("renderChart($0)",data);

This creates additional client-side state that must be synchronised and secured.

After (Vaadin 25, server-side generated SVG):

javaCopy

intvalue=75;Elementbar=newElement("rect");bar.setAttribute("x","0");bar.setAttribute("y","0");bar.setAttribute("width",String.valueOf(value));bar.setAttribute("height","20");bar.setAttribute("fill","#4caf50");Elementsvg=newElement("svg");svg.setAttribute("width","100");svg.setAttribute("height","20");svg.appendChild(bar);getElement().appendChild(svg);

The visualisation follows the server state directly. There is no duplicate logic and no hidden client-side code.

Example 3: Representation of a Formula with MathML

Before (text or rendered graphic):

javaCopy

Spanformula=newSpan("E = mc^2");

The formula’s semantic meaning is lost, as are its accessibility and machine-evaluability.

After (Vaadin 25, MathML):

javaCopy

Elementmath=newElement("math");Elementmrow=newElement("mrow");mrow.appendChild(newElement("mi").setText("E"));mrow.appendChild(newElement("mo").setText("="));mrow.appendChild(newElement("mi").setText("m"));mrow.appendChild(newElement("mo").setText("·"));Elementmsup=newElement("msup");msup.appendChild(newElement("mi").setText("c"));msup.appendChild(newElement("mn").setText("2"));mrow.appendChild(msup);math.appendChild(mrow);getElement().appendChild(math);

The formula is now semantically correct, accessible, and clearly structured – generated directly from Java.

These examples make it clear how Vaadin 25 can be used to implement technical visualisations without additional front-end dependencies. The Element API thus becomes a targeted tool for controlled, server-side presentation – exactly where classic components are no longer sufficient.

]]>ServersideVaadinhttps://svenruppert.com/posts/the-importance-of-ui-in-import-processes/Mon, 09 Feb 2026 15:15:33 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/the-importance-of-ui-in-import-processes/Why an import needs a UI at all Import functions are often treated as purely technical details in applications. Data is read in, processed and then made available – ideally without further interaction. In practice, however, an import is rarely an invisible process. It marks a transition between existing system states, between old and new data, between trust and control. This is exactly where the need for a user interface arises.<![CDATA[

Why an import needs a UI at all

Import functions are often treated as purely technical details in applications. Data is read in, processed and then made available – ideally without further interaction. In practice, however, an import is rarely an invisible process. It marks a transition between existing system states, between old and new data, between trust and control. This is exactly where the need for a user interface arises.

In the URL shortener, the import is not designed to run in the background; it is an explicit, visible process. The UI does not take on the role of a technical decision-maker, but that of a transparent mediator. It creates a clearly demarcated space in which an import can occur without immediately modifying existing data. This decoupling alone justifies its own import interface.

The current state of the source code is available on

GitHub:https://github.com/svenruppert/url-shortener orhttps://3g3.eu/url

1. Why an import needs a UI at all
2. The entry point in the application
3. The import dialog as a closed workspace
4. Implementation of the ImportDialog in Vaadin
5. File upload: Transport instead of interpretation
6. Validation as UI state change
7. Presentation of results without interpretation
8. Action Control and Apply Logic

Importing is always associated with uncertainty. Even if the data format is known, it remains unclear how incoming datasets interact with existing datasets. Are there any overlaps? Are conflicts arising? Are individual entries incomplete or invalid? These questions cannot be answered by a technical process alone, but require visibility. The UI makes this intermediate state tangible without prematurely evaluating it.

Instead of treating the import as an atomic step, it is understood as a state in the URL shortener. The user interface displays this state, records it, and prevents it from being overlooked and inadvertently entered into a production database. This makes the import process controlled, even if the UI itself does not control the content.

The clear distribution of roles is important here. The user interface does not interpret data, validate content, or make decisions about its technical correctness. Their task is exclusively to make the current technical status of the import visible. This deliberate restraint is crucial, as it prevents UI and import logic from being conflated.

So the import doesn’t need a UI because it’s complex, but because it’s a transition. The surface acts as a boundary between file and system, between potential changes and existing persistence. It gives the import a place, a time and a clear context – and that’s exactly where its raison d’être lies.

The entry point in the application

The import does not start automatically in the URL shortener; it only starts via a deliberately designed entry point in the application. It is where users manage existing short URLs and view the current database. This clearly positions the import as an exceptional act rather than part of the everyday processing flow.

Entry is done via an explicit action that opens a modal dialogue. This action only creates a new instance of the import dialogue and gives it the necessary dependencies. Further logic is deliberately not provided at this point. By opening the dialogue, the user leaves the application’s normal working context and enters a clearly demarcated area where the import is fully encapsulated.

The modal character of the dialogue fulfils an important function here. It indicates that the import is not a background process running in parallel, but a coherent process with its own state. As long as the dialogue is open, the rest of the UI state remains unchanged. There is no implicit update and no automatic data transfer.

From the user guidance perspective, this creates a clear-cut. The import is not understood as an extension of the table view, but as a temporary workspace. This separation prevents import actions from being mixed with regular editing steps. At the same time, it creates a mental orientation: everything that happens within this dialogue belongs to the import, and nothing beyond it.

From a technical standpoint, the UI assumes no further responsibility at this point. The entry point does not initiate validation or server communication; instead, it delegates the entire import process to the dialogue itself. Only when the user consciously uploads a file does the import process begin.

javaCopy

ButtonimportButton=newButton("Import");importButton.addClickListener(event->{ImportDialogdialog=newImportDialog(urlShortenerClient,()->{refreshGrid();});dialog.open();});

An excerpt from the corresponding view illustrates this principle. The button opens the import dialogue and optionally passes a callback, allowing a return to the normal UI context after a successful import.

The code makes it clear that the entry point merely instantiates and opens the dialogue. There is no pre-check or interaction with the server or import APIs. The callback is used only for downstream view updates and is not part of the import process itself.

Thus, the entry point defines not only where the import starts, but also how it is perceived. It is a clear turning point in the context of use and forms the basis for all subsequent steps in the import dialogue.

The import dialog as a closed workspace

When the import dialogue opens, the application deliberately switches to a well-defined working mode. The dialogue is not intended as a mere overlay, but as a standalone UI space with its own logic, state, and clearly defined boundaries. Everything that happens within this dialogue belongs exclusively to the import and has no effect on the existing database until the process is explicitly completed.

This partitioning is a central design element. While the dialogue is open, the rest of the application remains frozen in its previous state. Tables, filters or active selections are neither updated nor influenced. This creates a clear separation between ongoing work and the import process, providing both technical and mental orientation.

The import dialogue is implemented as a standalone Vaadin component. It is fully initialised when opened and has no implicit return channel to the calling view. This decoupling allows the dialog to manage its internal state independently. Neither when opened nor during the interaction is it assumed that import data already exists. The initial state is always neutral and empty.

Structurally, the dialogue is divided into distinct sections, each representing a specific phase of import. However, these areas are not implemented as step-by-step wizards; instead, they coexist within a common framework. This keeps the dialogue fully visible at all times and avoids implicit transitions that could obscure the actual state.

It is particularly important that the dialogue does not make a technical interpretation of the import data. It simply provides UI surfaces where results are displayed as soon as the server delivers them. Whether these spaces remain empty or are filled depends solely on the current import state, not on assumptions or expectations of the UI.

The control elements of the dialogue also follow this principle. Actions such as validation or application of the import are not permanently available, but are tied to the internal state of the dialogue. As long as there is no corresponding import data, the dialogue remains inactive. The UI does not enforce an order or progression, but only reacts to clearly defined state changes.

The import dialogue thus acts as a controlled container for a potentially critical operation. It fully encapsulates the import, makes its current status visible, and prevents incomplete or unchecked data from taking effect unnoticed. This clear boundary underpins all subsequent steps in the import process and explains why the dialogue was deliberately designed as a closed workspace.

Implementation of the ImportDialog in Vaadin

The import dialogue in the URL shortener clearly shows how little “framework magic” it takes to build a complete, state-driven UI with Vaadin Flow. The entire functionality is built from a few easy-to-understand building blocks: a dialogue container, standard components such asupload,grid,tabs, andbutton, and clear state variables that carry the dialogue throughout the import process. The result is an interface that treats the import as a separate workspace while keeping the source code manageable.

javaCopy

publicfinalclassImportDialogextendsdialogimplementsHasLogger{privatefinalURLShortenerClientclient;privatefinalUploadupload=newUpload();privatebyte[]zipBytes;privatefinalButtonbtnValidate=newButton("Validate");privatefinalButtonbtnApply=newButton("Apply Import");privatefinalButtonbtnClose=newButton("Close");privatefinalDivapplyHint=newDiv();privatestringstagingId;

Getting started is already remarkably simple. Dialog is a normal Java class that extendsDialogue. This means the UI is not a declarative construct or a template; it can be fully traced in Java code. At the same time, the dialogue deliberately remains stateful: it holds both the uploaded ZIP bytes and thestagingId generated by server-side validation.

With this structure, it’s already clear how Vaadin Flow works here: Components are objects, and the dialogue has them like ordinary fields. The UI is thus automatically addressable without requiring “UI IDs” or bindings. At the same time, the import state is not kept externally; it belongs precisely to the dialogue in which it is made visible.

The constructor then shows the core of the Vaadin mechanics. With just a few lines, the dialogue is configured as a modal, resizable container. There is no separate configuration file and no DSL, but pure Java calls. This allows the import dialogue’s visual framework to be understood directly.

javaCopy

setHeaderTitle("Import");setModal(true);setResizable(true);setDraggable(true);setWidth("1100px");setHeight("750px");

Immediately afterwards, it becomes apparent how Vaadin Flow typically models interactions. The buttons are initially deactivated and are unlocked only via events. This initialisation is a central part of the state model. The dialogue always starts neutrally and does not force progress unless there is a ZIP file.

javaCopy

btnValidate.addThemeVariants(ButtonVariant.LUMO_PRIMARY);btnValidate.setEnabled(false);btnApply.addThemeVariants(ButtonVariant.LUMO_SUCCESS);btnApply.setEnabled(false);btnClose.addClickListener(_->close());btnValidate.addClickListener(_->validate());btnApply.addClickListener(_->applyImport());

The interaction of UI and file upload can also be read directly. Theupload will be limited to ZIP files and will have a maximum size. The key point here is the in-memory handler: once the file is successfully uploaded, the bytes are placed inzipBytes and the next step is unlocked by enablingbtnValidate. The transition from “upload available” to “validation possible” is thus a single, very understandable state change.

javaCopy

upload.setAcceptedFileTypes(".zip",APPLICATION_ZIP);upload.setMaxFiles(1);upload.setMaxFileSize(IMPORT_MAX_ZIP_BYTES);UploadHandlerinMemoryUploadHandler=UploadHandler.inMemory((metadata,bytes)->{StringfileName=metadata.fileName();longcontentLength=metadata.contentLength();logger().info("uploaded file: fileName: {} , contentLength {}",fileName,contentLength);zipBytes=bytes;logger().info("setting zipBytes..");btnValidate.setEnabled(true);});upload.setUploadHandler(inMemoryUploadHandler);

The actual UI structure is then created inbuildContent(). Here, too, the strength of Vaadin Flow becomes apparent: Layouts are components, and the dialogue is simply built by assembling components. Headings, upload area, preview summary, tabs, and a central content container are merged into a single VerticalLayout. The code reads like a description of the interface.

javaCopy

varroot=newVerticalLayout(newH3("Upload ZIP"),upload,newH3("Preview"),summary,applyRow,tabs,tabContent);root.setSizeFull();root.setPadding(false);renderTab(tabContent);returnroot;

For displaying results, the dialogue uses twogrid instances and switches between them via tabs. The decisive factor is that the tab interaction does not trigger any backend logic; it only updates the visible projection. In Vaadin Flow, switching is a single listener that empties the container and uses the appropriate combination of paging bar and grid.

javaCopy

tabs.addSelectedChangeListener(e->renderTab(tabContent));privatevoidrenderTab(VerticalLayoutcontainer){container.removeAll();container.setSizeFull();if(tabs.getSelectedTab()==tabInvalid){container.add(pagingInvalid,gridInvalid);container.expand(gridInvalid);}else{container.add(pagingConflicts,gridConflicts);container.expand(gridConflicts);}}

At this point, it becomes clear why the implementation remains so compact despite several states. Vaadin Flow provides the UI building blocks, and the dialogue connects them via a few, clearly defined state variables and listeners. The actual import process remains server-side. The UI only needs to know when a state is reached and how to display it.

This makes theImportDialog a good example of how Vaadin Flow reduces complexity: not by hiding logic, but by using a direct programming model in which UI components, events, and state converge into a single, easy-to-read Java codebase. Especially for a minimalist import function, a complete, robust interface can be created quickly without requiring additional framework layers.

File upload: Transport instead of interpretation

In the import dialogue, the actual import process does not begin with a technical decision, but with a purely technical step: uploading a ZIP file. This moment marks the transition from an empty, neutral dialogue state to a potentially processable import without already making any content evaluation.

The upload component is firmly anchored in the dialogue and visible from the beginning. It deliberately serves as the first point of interaction within the closed work area. However, their task is clearly limited. The upload is used exclusively to receive a file and make its content temporarily available in the UI context. Neither the structure nor the semantics of the data contained play a role at this point.

This attitude is directly reflected in the source code. The upload is configured to accept exactly one ZIP file, and the maximum file size must not be exceeded. These restrictions do not serve the purpose of technical validation; they serve only the technical validation of the UI process.

javaCopy

upload.setAcceptedFileTypes(".zip",APPLICATION_ZIP);upload.setMaxFiles(1);upload.setMaxFileSize(IMPORT_MAX_ZIP_BYTES);

Once a file is selected and uploaded, an in-memory upload handler processes it. The dialogue saves the complete contents of the ZIP file as a byte array in an internal field. At this point, there is no check to verify whether the file contains import data or is structured correctly. In the UI, the file is initially a binary block.

javaCopy

UploadHandlerinMemoryUploadHandler=UploadHandler.inMemory((metadata,bytes)->{zipBytes=bytes;btnValidate.setEnabled(true);});upload.setUploadHandler(inMemoryUploadHandler);

These few lines mark a crucial UI state change. Upon successful upload completion, the validation button is enabled. That’s all that happens at this point. The upload alone does not trigger any server communication and does not create an import state. It merely signals that there is now enough information to trigger the next user step.

Errors during uploads are also dealt with exclusively from a technical perspective. If a file is rejected, for example, due to an incorrect data type or a file size that is too large, the UI responds with a corresponding notification. However, there is still no evaluation of the file’s content.

javaCopy

upload.addFileRejectedListener(event->{StringerrorMessage=event.getErrorMessage();Notificationnotification=Notification.show(errorMessage,5000,Notification.Position.MIDDLE);notification.addThemeVariants(NotificationVariant.LUMO_ERROR);});

The upload section of the dialogue thus clarifies a central design principle: The user interface does not interpret imported data. It merely ensures that a file is received in a technically correct manner and transfers this state into a clearly recognisable next possibility for action.

Only by clicking the “Validate” button does the dialogue leave this purely technical state of preparation. The file upload thus serves as the necessary, but deliberately content-free, basis for all subsequent steps of the import process.

Validation as UI state change

By clicking on the “Validate” button, the import dialogue leaves its purely preparatory state for the first time. Up to this point, only technical requirements were created: a file was uploaded and stored in memory without its content being interpreted or further processed. Thevalidate() method now marks the clear transition from a passive UI state to a state-changing step that makes the import visible for the first time.

From a user interface perspective, validation is not a technical review process, but a coordinated flow of multiple UI updates, all triggered by a single user action. The dialogue deliberately assumes no responsibility for the content. It asks the server to verify the uploaded ZIP content and processes only the technical response it receives.

The introduction to the method is correspondingly defensive. First, the dialogue state checks whether a ZIP file exists. If this is missing, the UI displays a brief notification and cancels the process. At this point, no exception is propagated, and no internal state is changed. The validation is thus clearly tied to a previous, explicit user action.

javaCopy

if(zipBytes==null||zipBytes.length==0){Notification.show("No ZIP uploaded.",2500,Notification.Position.TOP_CENTER);return;}

Only when this prerequisite is fulfilled is the actual validation step triggered. The dialogue passes the complete contents of the ZIP file to the client and calls the dedicated server endpoint for import preview. For the UI, this call is a black box. It does not know the internal validation rules or the criteria by which entries are classified as new, conflicting, or invalid.

javaCopy

StringpreviewJson=client.importValidateRaw(zipBytes);

The server’s response is received in its entirety as a JSON string. Instead of converting this into a fixed data model, the UI extracts the individual values relevant to representing the import state. This includes the generatedstagingId, as well as the counts of new, conflicting, and invalid entries. These values are immediately visible in the interface and constitute the first concrete import state displayed by the dialogue.

javaCopy

this.stagingId=extractJsonString(previewJson,"stagingId");intnewItems=extractJsonInt(previewJson,"newItems",0);intconflicts=extractJsonInt(previewJson,"conflicts",0);intinvalid=extractJsonInt(previewJson,"invalid",0);

With these values set, the character of the dialogue changes noticeably. The previously empty preview area is populated, and the import receives an identity for the first time: thestagingId. Nevertheless, the UI remains strictly descriptive. It does not evaluate whether the figures are plausible or in a certain relationship to each other. It only shows the current technical status.

At the same time, the dialogue reads the result lists directly from the server response. Iterators are used to read JSON arrays for conflicts and invalid entries, and to convert them into UI-internal row objects. These are then assigned to the corresponding grids. Here, too, there is no interpretation. If an array is empty or non-existent, the tables remain empty – a state that the UI treats as completely valid.

javaCopy

for(Stringobj:newItemsArrayIterator(r,"conflictItems")){Map<String,String>m=JsonUtils.parseJson(obj);conflictRows.add(ConflictRow.from(m));}

After the grids are filled, paging information is updated, and the associated UI components are synchronised. This step is also performed exclusively based on the previously determined result lists, without any additional feedback to the server.

With the completion of thevalidate() method, the import dialogue is in a new, stable state. The uploaded data is validated on the server side; the results are visible, and the dialogue can now decide which further actions to offer the user. Note that validation itself does not trigger an import. It only changes the UI state and creates transparency about what a subsequent import would do.

Thevalidate() method is thus the central linchpin of the entire import dialogue. It combines the upload and presentation of results without making technical decisions itself. In this role, it becomes the core of the UI-controlled import process.

Presentation of results without interpretation

After validation, the import dialogue is in a state where concrete results are available for the first time. However, these results are not interpreted, weighted, or further processed; they are only presented. The last chapter describes exactly this moment: the transformation of raw information delivered on the server side into visible UI structures – without the user interface drawing its own conclusions.

Central to this chapter is the realisation that the dialogue lacks a technical view of conflicts or invalid entries. He knows neither their significance nor their effects. Instead, it performs a purely mechanical task: it takes lists of results and renders them into prepared UI components.

This projection is done via two separate tables, each associated with a specific result type. Conflicts and invalid entries are deliberately not presented together; they are kept in separate contexts. The dialogue offers two tabs for this purpose, which the user can switch between. However, this change only changes the view of the data, not its content or state.

The technical basis of this representation consists of two grid components, which are already fully configured when the dialogue is set up. Columns, widths, and display logic are fixed before it is even known whether data will ever be displayed. The grids thus exist independently of the existence of results and represent a stable projection surface.

javaCopy

gridConflicts.addColumn(ConflictRow::shortCode).setHeader("shortCode").setAutoWidth(true).setFlexGrow(0);gridConflicts.addColumn(ConflictRow::diff).setHeader("diff").setAutoWidth(true).setFlexGrow(0);gridConflicts.addColumn(ConflictRow::existingUrl).setHeader("existingUrl").setAutoWidth(true);gridConflicts.addColumn(ConflictRow::incomingUrl).setHeader("incomingUrl").setAutoWidth(true);

These grids are filled not through a traditional data model, but via direct iteration over JSON fragments in the server response. The UI reads each object from its corresponding array and converts it into a simple line representation. These row objects contain exactly the fields that should be displayed, no more and no less.

javaCopy

for(Stringobj:newItemsArrayIterator(r,"conflictItems")){Map<String,String>m=JsonUtils.parseJson(obj);conflictRows.add(ConflictRow.from(m));}

It is noteworthy that the dialogue makes no assumptions about the number of expected entries or about which fields must be present. If an array is empty or cannot be read, the associated table remains empty. This state is not treated by the UI as an error; it is considered a valid result of validation.

The tab control also follows this principle of neutrality. The currently selected tab only determines which table is visible. When switching, no data is reloaded, and no states are changed. The UI only shows a different section of the same import state.

javaCopy

tabs.addSelectedChangeListener(e->renderTab(tabContent));

The presentation of results is supplemented by a simple paging component that operates only on results already loaded. It provides a better overview of large datasets but is fully local and does not execute any additional server queries. Here, too, no content filter is applied; paging is purely representation-oriented.

The interplay of grids, tabs, and paging results in a deliberately restrained user interface. It shows what’s there and doesn’t show anything that hasn’t been clearly delivered. Neither is there an attempt to compensate for missing data, nor are implicit assumptions made about the “actual” state of the import.

This makes it clear that the result displayed in the import dialogue is not an analysis tool, but a mirror. It reflects the state delivered by the server exactly and defers any further evaluation to subsequent steps in the import process.

Action Control and Apply Logic

After uploading, validating, and displaying the results, one central question remains in the import dialogue: Can the import actually be used? The answer to this question is not implicit or automatic. Instead, it is completely controlled by the UI state.

The basis of this control is not an additional server call, but the already known import state. All information relevant to the decision is already available to the UI. TheupdateApplyState() method serves as a central hub that interprets the state and translates it into concrete UI activations or deactivations.

The starting point is a strictly defensive default state. Regardless of what happened before, the Apply button will be disabled first. The UI never assumes that an import is automatically applicable. Only when all conditions are explicitly fulfilled is this state lifted.

javaCopy

btnApply.setEnabled(false);btnApply.setText("Apply Import");

The first hard test point is the presence of astagingId. Without this identifier, there is no valid import context from a UI perspective. Even if the data is already displayed, the import remains inapplicable until a server-side confirmed staging state is reached. The UI does not treat this case as an error, but as an incomplete state.

Then, the two result dimensions that have already been made visible in the previous chapters are considered: invalid entries and conflicts. Invalid entries generally block the import. As soon as at least one invalid record exists, the Apply function remains deactivated, and the dialogue explicitly communicates this state via a hint text. The UI does not force a correction; it simply makes it clear that an import is not possible under these conditions.

Conflicts, on the other hand, are treated differently. From a UI perspective, they do not represent an absolute exclusion but rather a deliberate decision-making process. The dialogue includes a checkbox for this purpose, allowing the user to specify whether conflicting entries should be skipped during import. Only with this explicit consent is the import released despite existing conflicts.

This differentiation is directly evident in the interplay of the checkbox, information text and apply button. Activating or deactivating the checkbox immediately triggers a reassessment of the UI state without loading or recalculating any additional data. The UI responds only to the known state.

javaCopy

if(conflicts>0&&!chkSkipConflicts.getValue()){applyHint.setText("Apply disabled: "+conflicts+" conflict(s). Tick "Skipconflictsonapply" to proceed.");return;}

When the import is finally approved, not only does the button’s activation change, but its label does as well. This allows the UI to clearly signal the conditions under which the import will be executed. This visual notice is part of the action management and serves to ensure transparency, not to enforce professional rules.

By clicking “Apply Import”, the dialogue leaves its purely display and decision modes. Only at this point is another server call triggered, which actually applies the previously validated import. Up to this point, the UI has only managed states, displayed them and demanded decisions.

The action control thus forms the deliberate conclusion of the import dialogue. It bundles all previously built-up information and converts it into an explicit user decision. It is precisely this restraint – not to apply anything automatically, to imply anything – that makes dialogue a controlled and comprehensible tool within the application.

]]>JavaServersideVaadinhttps://svenruppert.com/posts/20224/Thu, 05 Feb 2026 16:50:15 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/20224/Export functions are often seen as a purely technical side task: one button, one download, done. In a Vaadin-based application, however, it quickly becomes apparent that exporting is much more than writing data to a file. It is a direct extension of the UI state, an infrastructural contract between frontend and backend, and a decisive factor for maintainability and predictability.<![CDATA[

Export functions are often seen as a purely technical side task: one button, one download, done. In a Vaadin-based application, however, it quickly becomes apparent that exporting is much more than writing data to a file. It is a direct extension of the UI state, an infrastructural contract between frontend and backend, and a decisive factor for maintainability and predictability.

1. Export from a UI point of view: more than a download button
2. Initial situation: functional export, but non-UI
3. Design Goal: Export as a Deterministic UI Workflow
4. Uniform responses as a prerequisite for clean UI logic
5. Filter logic as a common language between the grid and export
6. Download mechanics in Vaadin: Button ≠ Download
7. StreamResource: Export on demand instead of in advance
8. Paging boundaries as a protective mechanism
   1. Real JSON export from the running system
9. Effects on maintainability and comprehensibility
10. Conclusion

This article shows how a JSON-based export was deliberately designed as a UI-driven workflow in the URL shortener project. The focus is not on file formats or complex backend abstractions, but on the clean embedding of the export in a Vaadin Flow interface: filter coupling, download mechanics, paging boundaries and clear responsibilities between UI, client and server.

**The current source code can be found on GitHub underhttps://github.com/svenruppert/url-shortener orhttps://3g3.eu/url

Export from a UI point of view: more than a download button

In classic web applications, export is often thought of as an isolated API endpoint. From the perspective of a Vaadin UI, this consideration falls short. For the user, export is not a technical process but a consequence of the current UI state: the filters applied, the sorting, and the paging limits.

An export that ignores this coupling immediately leads to cognitive breaks. The display in the grid shows a certain amount of data, but the export provides something else – be it more, less or simply different data. This is exactly where it is determined whether an application is perceived as consistent.

The claim in the project was therefore clear: The export is not a special function, but a mirror of the UI state.

This decision shapes all further design steps – from the filter generation to the download mechanics to the structure of the export data itself.

Initial situation: functional export, but non-UI

Before the revision, an export was already technically possible. Data could be read on the server and returned to the client as JSON, ensuring the basic functionality was fulfilled. However, from a user interface perspective, this implementation introduced several structural issues that only became apparent upon closer inspection.

The response structures were inconsistent and required the client to interpret them in context. The meaning and interaction of HTTP status codes and response bodies were not explicitly defined; they were derived from implicit assumptions in the client code. At the same time, there was no clear connection between the export and the currently visible filters in the interface. From the UI’s perspective, it was only possible to trace the data that was actually exported indirectly. In addition, there was special logic for empty results or error cases that could not be derived consistently from the response itself but were distributed across several places in the client.

From Vaadin’s perspective, the export was not an integrated UI workflow but rather an isolated technical endpoint. The UI had to provide knowledge of special cases, status codes, and response formats that were not explicitly covered by a contractually defined framework. This state of affairs was also reflected in the test landscape: tests were often based on concrete string representations or complete JSON output rather than on clearly defined functional structures. Changes to filters, sorting or response formats therefore had to be followed up in several places and carried an increased risk of unintended side effects.

In short, the export worked technically but did not meet the requirements for a UI-enabled, traceable, and maintainable component in a Vaadin application.

Design Goal: Export as a Deterministic UI Workflow

The central goal of the redesign was not “more features”, but predictability. For the Vaadin UI, this means:

The export uses the same filters as the grid and thus reflects the current UI state. Paging limits are deliberately set and comprehensible for developers and users alike, so that the scope and character of the export remain clearly recognisable. Success, empty results and error cases are clearly distinguishable and can be handled in the UI without special logic. At the same time, the download behaves browser-compliantly and UI-stably without affecting the current UI state.

From the UI’s perspective, an export must not have its own state. He must not “think” anything, expand anything, or change anything implicitly. It is a snapshot of what the user sees – nothing more, nothing less.

Uniform responses as a prerequisite for clean UI logic

In a Vaadin application, API responses have an immediate effect on the UI code, as each response is typically translated directly into UI states, component logic, and user feedback. In contrast to purely client-side frontends, the UI logic is tightly coupled to server-side processing: Each response directly updates component state, enables or disables controls, and renders feedback to the user.

In this context, inconsistent response formats inevitably lead to complex if-else cascades in the UI code. Special treatments for seemingly trivial cases, such as “empty” exports or different error states, must be explicitly requested. The UI code starts by interpreting technical details of the API – such as certain HTTP status codes or the presence of individual JSON fields – instead of relying on clearly defined business signals. This not only increases the code complexity but also complicates the interface behaviour during extensions, making it harder to understand and more error-prone.

In the URL shortener project, this problem was solved by introducing an explicit and stable response structure. Regardless of whether an export record is empty or contains an error, the response always follows the same structure. HTTP status codes are still used to signal the rough outcome of a request, but they do not serve as the sole signifier. The actual technical information – such as the context, scope, and content of the export – is transmitted in full and consistently in JSON format.

A simplified, real export from the system illustrates this approach:

jsonCopy

{ "formatVersion":"1", "mode":"filtered", "exportedAt":"2026-02-05T11:28:54.582886239Z", "total":9, "items":[/*subjectrecords*/]}

This creates a clear and stable contract for the Vaadin UI. The UI code can rely on the fact that metadata such as mode, exportedAt, or total is always present and interpreted consistently. The interface no longer has to guess whether an export was successful or whether there are special cases. Instead, the process can be designed in a linear, deterministic way: metadata is evaluated, the scope is checked, and the user data is processed or reported back to the user.

This structure has far-reaching consequences for UI logic. Loading indicators, confirmation dialogs or error messages can be derived directly from the structured response, without additional special logic or context-dependent checks. This keeps the interface clear, predictable, and closely linked to the technical significance of the answer, rather than tied to technical special cases or implicit assumptions.

Filter logic as a common language between the grid and export

A crucial Vaadin-specific point is the reuse of the filter logic. There is no separate export filter in the project. Instead, the export is generated exclusively from the current UI state.

The SearchBar acts as the only source of truth:

javaCopy

publicUrlMappingListRequestbuildFilter(intpage,intsize){ UrlMappingListRequestreq=newUrlMappingListRequest(); req.setPage(page); req.setSize(size); req.setActiveState(activeState); req.setCodePart(codeField.getValue()); req.setUrlPart(urlField.getValue()); req.setFrom(from); req.setTo(to); req.setSort(sort); req.setDir(dir); returnreq;}

This Request object is used for both grid display and export. This guarantees:

Display and export thus produce identical results, since both are based on the same filter definitions. Changes to filters or collations automatically and consistently affect display and export without requiring additional code. At the same time, there are no hidden or implicit export parameters, so the export behaviour can be fully explained by the UI state.

From a maintenance perspective, this is a significant advantage: if you understand the UI, you understand the export.

Download mechanics in Vaadin: Button ≠ Download

A common mistake in Vaadin applications is trying to start a file download directly from a button click. Technically, this is problematic: a button click primarily triggers server-side logic, whereas a download is a resource from the browser’s perspective.

In Vaadin, a button click is primarily aserver-side UI event. The browser does not send a “classic” download request; instead, Vaadin processes the click via itsUI/RPC communication (server round-trip, event listener, component update). From the browser’s perspective, this isnot a normal navigation or resource retrieval. And that’s exactly why “button clicks → browser downloads file” is not reliable, because the browser typically only starts a download cleanly when it retrieves aresource (link/navigation) or submits a form – i.e. something that is perceived in the browser as a “real request for a file”.

The anchor (<a>) element solves this problem because it is astandard download target for the browser: it has an href attribute that points to a resource, and the download attribute signals to the browser: “This is a file”. In Vaadin, you bind this href to a StreamResource. This creates aseparate HTTP request when clicking the anchor, which is not part of the Vaadin UI event flow but rather an independent resource retrieval. Only at this moment is the StreamResource “pulled”, and the export contentis generated on demand.

In practice, this has three major advantages:

1. Browser compliance and reliability: The download is started via a mechanism that the browser natively supports. This reduces edge cases in which a download triggered by a UI event is blocked or behaves inconsistently (e.g., pop-up/download policies, timing, UI updates).
2. Decoupling from the UI lifecycle: The download occurs in a separate request. Even if Vaadin processes UI requests in parallel, if the user clicks on or rerenders the interface, the download can continue to run stably. This is especially important if export generation takes longer or is streamed.
3. Clean accountability: The button is purely UI/UX (icon, tooltip, permissions, enable/disable, visual feedback). The anchor is purely “transport” (browser download). The StreamResource is purely a “data supplier” (the export is generated only when needed). This separation makes the code more maintainable and reduces the side effects.

javaCopy

ButtonbtnExport=newButton(VaadinIcon.DOWNLOAD.create());btnExport.setTooltipText("Export current result set as ZIP");btnExport.addClickListener(e->   exportAnchor.getElement().callJsFunction("click"));

The actual download behaviour is in the anchor connected to a StreamResource:

javaCopy

StreamResourceexportResource=   newStreamResource("export.zip",()->{     UrlMappingListRequestfilter=         searchBar.buildFilter(1,chunkSize);     returnurlShortenerClient.exportAllAsZipDownload(filter);   });exportAnchor.setHref(exportResource);exportAnchor.getElement().setAttribute("download",true);

This pattern clearly separates the responsibilities: the UI interaction is limited to the button, which serves exclusively as a trigger for export. The browser download is triggered via the anchor element and is therefore treated as a regular resource request. Finally, the data is made available via the StreamResource, which only generates the export content when it is actually downloaded.

The export is only generated when the browser actually retrieves the resource – not when the user clicks on it.

StreamResource: Export on demand instead of in advance

The use of StreamResource is not a detail, but a deliberate architectural decision. The export is generated on demand while the browser reads the stream.

This has several advantages. On the UI side, the memory footprint remains low because the export does not need to be fully pre-generated and buffered. At the same time, the UI thread is not blocked because the data transfer occurs outside the regular UI lifecycle. The download can continue regardless of the current UI state, even if the user navigates or performs further actions during this time. If errors occur during stream generation, they can be propagated cleanly via a separate HTTP request without causing the UI state to become inconsistent.

The export is thus technically decoupled from the UI lifecycle, although it is logically triggered by the UI.

Paging boundaries as a protective mechanism

Another explicitly UI-related aspect of the export implementation is the deliberate limit on export quantity. The export uses the same chunkSize as the grid in the interface and is additionally limited by a fixed upper limit. This decision ensures that the export always remains within a clearly defined framework and can be derived directly from the current UI state.

From an architectural perspective, this limitation prevents the export from processing large amounts of data in an uncontrolled manner when a user triggers an export. Especially in Vaadin applications, where UI interactions are typically synchronous with server-side logic, this protective measure is crucial. It reduces the risk of heavy memory loads, long runtimes, or blocking operations that could negatively impact other users or the entire server.

At the same time, the paging boundary conveys a clear technical semantics to the outside world. The export is deliberately defined as an image of the currently visible result set. It mirrors exactly what the user sees in the grid, including filtering, sorting, and paging configurations. This does not imply a claim to completeness, as is typically associated with a backup.

This clarity is particularly relevant for user expectations. The export does not provide a complete system print or a historically complete data set, but a specifically selected excerpt. The limitation makes this character explicit and prevents misinterpretations, such as assuming that an export can fully restore the system.

From a maintenance and operations perspective, the paging boundary also serves as a natural safety line. It forces us to consciously design export scenarios and, if necessary, to provide separate mechanisms for backups or mass data withdrawals. As a result, the export remains a controllable UI tool and does not insidiously become an infrastructural backdoor for unlimited data queries.

In summary, limiting export volume is not a technical constraint but a deliberate design decision. It combines UI state, user expectations and system stability into a consistent overall picture and underlines once again that the export in the URL shortener is understood as a UI-driven result set – and expressly not as a substitute for a backup.

Real JSON export from the running system

The architectural decisions described above are particularly well understood from a real export of the running system. The following JSON export was generated directly from the Vaadin interface and represents a specific UI state at a defined point in time.

Even at the top level, the export contains all the necessary contextual information to enable independent classification. The formatVersion field explicitly defines the export format version, providing a stable foundation for future extensions. Changes to the internal data model do not automatically propagate to the export contract, provided the version limit is respected.

The field mode is deliberately chosen to speak. The filtered value makes it unmistakably clear that this is not a complete data deduction, but a result set restricted by UI filters. This information is crucial because it prevents the export from being mistakenly interpreted as a backup. The export does not capture the entire system state; it only includes the section the user has seen in the grid.

With exportedAt, the exact time of snapshot creation is recorded. The export thus clearly refers to a defined system state. Later changes to individual data records are deliberately not included and can be clearly delineated on the basis of this time stamp. This context is supplemented by the total field, which indicates the number of exported data records and enables a quick plausibility check without analysing the actual user data.

The actual technical data is located exclusively in the items array. Each entry describes a single URL-mapping dataset, including subject-relevant properties such as shortCode, originalUrl, and active, as well as temporal attributes createdAt and, optionally, expiresAt. It is notable that these objects contain no UI or export-specific metadata. They are deliberately reduced to the technical core and could also come from other contexts in the same form.

It is precisely this clear separation between top-level metadata and functional user data in the items array that makes the export an explainable artefact in itself. Even without knowledge of the internal code or the Vaadin interface, it is possible to determine when the export was created, under what conditions, its scope, and where the actual technical data begins.

The real export thus confirms the design goals described above. It is reproducible, rich in context and clearly recognisable as a UI-driven result set. Instead of merely transporting data, it also conveys its meaning and context of creation – a property that is crucial for maintainability, analysis and long-term further processing.

Effects on maintainability and comprehensibility

The tight coupling between the export and the UI state ensures behaviour that is predictable for developers and users alike. The export follows the same rules as the grid display and contains no hidden special paths or implicit deviations. As a result, the export automatically evolves with the UI: any adjustment to filters, sorting, or paging mechanisms has a consistent effect on both paths without requiring additional synchronisation code.

From a developer’s perspective, this architecture significantly reduces cognitive load. There is no separate mental model space for exporting, as its behaviour can be completely derived from the known UI state. If you understand the grid and its filter logic, you automatically understand the export. This not only simplifies onboarding new developers but also reduces the risk of unintentional inconsistencies during refactorings or functional enhancements.

Testability also benefits directly from this clarity. Since the export has no state and relies on stable request and response structures, it can be tested in isolation. Tests can be run with specific filter combinations and validate the resulting exports without simulating the entire UI or complex interaction sequences. At the same time, UI tests remain lean because they can focus on correctly generating the filter state.

In the long term, this structure improves the maintainability of the overall system. Changes to the UI do not introduce hidden side effects in the export, and conversely, further development of the export does not require parallel adjustments elsewhere. The risk of divergent logic paths between display and export is not only reduced but systematically eliminated.

In summary, the close integration of UI state and export logic ensures that export is not a special case in the system. It becomes a transparent, explainable, long-term, and maintainable component of the application that fits seamlessly into the existing Vaadin architecture.

Conclusion

The export in the URL shortener is not an isolated API endpoint, but an integral part of the Vaadin UI architecture. It follows the same rules as the grid, uses the same filters and respects the same boundaries.

Vaadin Flow applications in particular show that a cleanly integrated export is less a question of the file format – and much more a question of clear responsibilities, explicit contracts and a consistently conceived UI workflow.

]]>JavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-extracting-components-part-2/Mon, 22 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-extracting-components-part-2/What has happened so far In the first part, the focus was deliberately on the user interface’s structural realignment. The previously grown, increasingly monolithic OverviewView was analysed and specifically streamlined by outsourcing key functional areas to independent UI components. With the introduction of the BulkActionsBar and the SearchBar, clearly defined building blocks were created, each assuming a specific responsibility and freeing the view from operational details.<![CDATA[

What has happened so far

In the first part, the focus was deliberately on the user interface’s structural realignment. The previously grown, increasingly monolithic OverviewView was analysed and specifically streamlined by outsourcing key functional areas to independent UI components. With the introduction of the BulkActionsBar and the SearchBar, clearly defined building blocks were created, each assuming a specific responsibility and freeing the view from operational details.

This refactoring was not a cosmetic step but a conscious investment in the application’s long-term maintainability. By separating presentation, interaction, and logic, a modular foundation was created that is not only easier to test but also significantly simplifies future extensions. The OverviewView transformed from a function-overloaded central element into an orchestrating instance that brings components together instead of controlling their internal processes.

The second part now builds upon this foundation. While Part 1 highlighted the motivation, goals, and initial extractions, the focus now shifts consistently to the new structure of the OverviewView itself: how its role has changed, how event flows have been simplified and how the interaction of the extracted components leads to a clearer, more stable architecture.

The source code for this article can be found on GitHub at:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-11

Here is a screenshot of the current development status from the user’s perspective.

The focus is on dividing the previously monolithic OverviewView into clearly defined, reusable components such as the BulkActionsBar and the SearchBar. This separation improves maintainability, increases clarity and makes it easier to test individual functional areas. At the same time, this will pave the way for further UI optimisations and interaction patterns to be implemented gradually in the coming days.

New structure of the OverviewView

After removing the subcomponents, the new OverviewView appears in a much more streamlined, clearer form. While previously a large number of elements, event handlers and logic fragments were spread across the entire class, the View is now limited to a few, clearly defined tasks: the initialisation of the core components, the setup of the grid and the orchestration of the interactions between the SearchBar, BulkActionsBar and the backend.

This new structure follows a simple but powerful principle: the OverviewView focuses on bringing the components together rather than on how they work internally. It defines which building blocks are displayed, how they work together, and when they need to be updated. The internal structure of the individual elements – whether it be the handling of filter changes, the management of bulk actions, or the technical logic of the DataProvider – lies entirely within the respective components.

This clear layout makes the OverviewView look much tidier. The previously lush sections, with interspersed event listeners, complex UI layouts, and manual error handling, have largely disappeared, either merged into outsourced components or removed. They are replaced by short, easy-to-understand methods that control the flow of the view and connect its various building blocks.

This reduction not only makes the OverviewView easier to maintain but also easier to expand. New functions can be integrated at appropriate points without risking damage to existing logic. At the same time, an easily testable structure is created, as the dependencies are clearly defined and the responsibilities are clearly separated.

A central element of the new structure is the initialisation of the view, which is now clearly structured and largely free of embedded logic. This new distribution of roles can already be clearly seen in the constructor:

kotlinCopy

publicOverviewView(){setSizeFull();setPadding(true);setSpacing(true);add(newH2("URL Shortener – Overview"));initDataProvider();varpagingBar=newHorizontalLayout(prevBtn,nextBtn,pageInfo,btnSettings);pagingBar.setDefaultVerticalComponentAlignment(CENTER);HorizontalLayoutbottomBar=newHorizontalLayout(newSpan(),pagingBar);bottomBar.setWidthFull();bottomBar.expand(bottomBar.getComponentAt(0));bottomBar.setAlignItems(CENTER);VerticalLayoutcontainer=newVerticalLayout(searchBar,bottomBar);container.setPadding(false);container.setSpacing(true);container.setWidthFull();add(container);add(bulkBar);add(grid);configureGrid();addListeners();addShortCuts();try(var_=withRefreshGuard(false)){searchBar.setPageSize(25);searchBar.setSortBy("createdAt");searchBar.setDirValue("desc");}catch(Exceptione){thrownewRuntimeException(e);}}

Here it becomes clear: The OverviewView only instantiates its components, inserts them into a layout and delegates all details to specialised classes. Neither filter logic nor bulk operations are included here – the view only orchestrates.

Another example of the streamlined structure is the way selection is handled in the grid. In the past, there was extensive logic for buttons, states and actions here; today, the view only controls the visibility and status of the BulkActionsBar:

kotlinCopy

grid.addSelectionListener(event->{varall=event.getAllSelectedItems();booleanhasSelection=!all.isEmpty();bulkBar.setVisible(hasSelection);if(hasSelection){intcount=all.size();Stringlabel=count==1?"link selected":"links selected";bulkBar.selectionInfoText(count+" "+label+" on page "+currentPage);}else{bulkBar.selectionInfoText("");}bulkBar.setButtonsEnabled(hasSelection);});

So the view only handles showing or hiding the BulkActionsBar. The actual logic for executing the actions lies entirely in the component itself.

The refresh mechanic is now clearly defined as well. Instead of repeating refresh calls in many places in the code, thesafeRefresh() method has been created, which centrally defines how updates are performed:

javaCopy

publicvoidsafeRefresh(){logger().info("safeRefresh");if(!suppressRefresh){logger().info("refresh");dataProvider.refreshAll();}}

This design not only makes updating cleaner, but also prevents duplicate refreshes and unwanted infinite loops.

The new clarity is also reflected in the configuration of the grid, which remains complex in terms of content, but is clearly delineated and outsourced to its own methods:

xmlCopy

private void configureGrid() { logger().info("configureGrid.."); grid.addThemeVariants(GridVariant.LUMO_ROW_STRIPES, GridVariant.LUMO_COMPACT); grid.setHeight("70vh"); grid.setSelectionMode(Grid.SelectionMode.MULTI); configureColumShortCode(); configureColumUrl(); configureColumCreated(); configureColumActive(); configureColumExpires(); configureColumActions(); grid.addItemDoubleClickListener(ev -> openDetailsDialog(ev.getItem())); grid.addItemClickListener(ev -> { if (ev.getClickCount() == 2) openDetailsDialog(ev.getItem()); }); GridContextMenu<ShortUrlMapping> menu = new GridContextMenu<>(grid); menu.addItem("Show details", e -> e.getItem().ifPresent(this::openDetailsDialog)); menu.addItem("Open URL", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().open(m.originalUrl(), "_blank"))); menu.addItem("Copy shortcode", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", m.shortCode()))); menu.addItem("Delete...", e -> e.getItem().ifPresent(m -> confirmDelete(m.shortCode())));}

This structure makes it clear which areas of the view have which tasks. The multi-outsourced logic ensures that each method fulfils a clearly defined function.

Simplified event pipeline after refactoring

Refactoring has significantly reduced this complexity. The event pipeline now follows a linear, predictable flow: users interact with the UI components, which trigger clearly defined actions; the OverviewView responds with manageable control signals and finally updates the grid via a uniform refresh mechanism. Critical is the introduction ofsafeRefresh() andwithRefreshGuard(), which prevent unnecessary or recursive updates.

The result is an architecture in which events are no longer propagated across the view, but run along a handful of defined interfaces in a structured manner. This is already clearly visible in the central listener setup of the OverviewView:

javaCopy

privatevoidaddListeners(){ComponentUtil.addListener(UI.getCurrent(),MappingCreatedOrChanged.class,_->{logger().info("Received MappingCreatedOrChanged -> refreshing overview");refreshPageInfo();safeRefresh();});grid.addSelectionListener(event->{varall=event.getAllSelectedItems();booleanhasSelection=!all.isEmpty();bulkBar.setVisible(hasSelection);if(hasSelection){intcount=all.size();Stringlabel=count==1?"link selected":"links selected";bulkBar.selectionInfoText(count+" "+label+" on page "+currentPage);}else{bulkBar.selectionInfoText("");}bulkBar.setButtonsEnabled(hasSelection);});btnSettings.addClickListener(_->newColumnVisibilityDialog<>(grid,columnVisibilityService).open());prevBtn.addClickListener(_->{if(currentPage>1){currentPage--;refreshPageInfo();safeRefresh();}});nextBtn.addClickListener(_->{intsize=Optional.ofNullable(searchBar.getPageSize()).orElse(25);intmaxPage=Math.max(1,(int)Math.ceil((double)totalCount/size));if(currentPage<maxPage){currentPage++;refreshPageInfo();safeRefresh();}});}

This is where the new clarity becomes apparent: External events such asMappingCreatedOrChanged trigger a targeted refresh; the grid selection only affects the BulkActionsBar; the paging buttons only control page and refresh. The complex logic from earlier days is clearly divided.

The keyboard shortcuts also integrate seamlessly into this simplified pipeline and use the encapsulated bulk logic:

javaCopy

privatevoidaddShortCuts(){varcurrent=UI.getCurrent();current.addShortcutListener(_->{if(!grid.getSelectedItems().isEmpty()){bulkBar.confirmBulkDeleteSelected();}},Key.DELETE);}

On the SearchBar side, event processing is also greatly relieved and follows a clear pattern. Changes to filters or settings always lead to the OverviewView and its uniform refresh mechanism:

javaCopy

activeState.addValueChangeListener(_->{holdingComponent.setCurrentPage(1);holdingComponent.safeRefresh();});pageSize.addValueChangeListener(e->{holdingComponent.setCurrentPage(1);holdingComponent.setGridPageSize(e.getValue());holdingComponent.safeRefresh();});resetBtn.addClickListener(_->{try(var_=withRefreshGuard(true)){resetElements();holdingComponent.setCurrentPage(1);}catch(Exceptione){thrownewRuntimeException(e);}});

The use ofwithRefreshGuard(true) in combination withsafeRefresh() ensures that certain internal state changes do not immediately trigger a cascade refresh, but are triggered in a controlled, conscious manner.

Improvements in the MultiAliasEditorStrict

The MultiAliasEditorStrict is a central UI element in the URL shortener, allowing you to edit and manage multiple alias variants of a shortlink. During the refactoring, this editor was also revised to better integrate with the new component architecture and, at the same time, improve the user experience. Many of the original challenges resulted from tight integration with the OverviewView and from a poorly structured internal logic that had grown over the course of development.

One of the most essential innovations concerns the consistency of his behaviour. The MultiAliasEditorStrict is implemented as a standalone, clearly focused UI element that is solely responsible for capturing and validating aliases:

xmlCopy

public class MultiAliasEditorStrict extends VerticalLayout { private static final String RX = "^[A-Za-z0-9_-]{3,64}$"; private final Grid<Row> grid = new Grid<>(Row.class, false); private final TextArea bulk = new TextArea("Aliases (comma/space/newline)"); private final Button insertBtn = new Button("Take over"); private final Button validateBtn = new Button("Validate all"); private final String baseUrl; private final Function<String,Boolean> isAliasFree; Server check (true = free) public MultiAliasEditorStrict(String baseUrl, Function<String,Boolean> isAliasFree) { this.baseUrl = baseUrl; this.isAliasFree = isAliasFree; build(); }

It is already clear here that the editor has a well-defined task: it knows the previewbaseUrl prefix, manages its own UI elements, and provides anisAliasFreefunction to check for alias conflicts on the server side.

The structurinterface’s interface is entirely contained within thebuild() method. This is where text input, toolbar and grid are assembled:

javaCopy

privatevoidbuild(){setPadding(false);setSpacing(true);bulk.setWidthFull();bulk.setMinHeight("120px");bulk.setValueChangeMode(ValueChangeMode.LAZY);bulk.setClearButtonVisible(true);bulk.setPlaceholder("e.g.\nnews-2025\npromo_x\nabc123");insertBtn.addClickListener(_->parseBulk());validateBtn.addClickListener(_->validateAll());vartoolbar=newHorizontalLayout(insertBtn,validateBtn);toolbar.setSpacing(true);configureGrid();add(bulk,toolbar,grid);}

This makes the user interface clear: First, aliases are entered in the bulk field, then transferred to the grid via “Take over” and finally checked via “Validate all”.

The MultiAliasEditorStrict has also been improved in terms of structure and internal architecture. Instead of distributing logic fragments across validation, synchronisation, and UI updates in an unstructured way, these areas were clearly separated and moved to dedicated methods. This is particularly evident in the grid structure and in the validation logic.

The grid itself displays the alias lines, a preview and the status in a compact form:

javaCopy

privatevoidconfigureGrid(){grid.addComponentColumn(row->{vartf=newTextField();tf.setWidthFull();tf.setMaxLength(64);tf.setPattern(RX);tf.setValue(Objects.requireNonNullElse(row.getAlias(),""));tf.setEnabled(row.getStatus()!=Status.SAVED);tf.addValueChangeListener(ev->{row.setAlias(ev.getValue());validateRow(row);grid.getDataProvider().refreshItem(row);});returntf;}).setHeader("Alias").setFlexGrow(1);grid.addColumn(r->baseUrl+Objects.requireNonNullElse(r.getAlias(),"")).setHeader("Preview").setAutoWidth(true);grid.addComponentColumn(row->{varlbl=switch(row.getStatus()){caseNEW->"New";caseVALID->"Valid";caseINVALID_FORMAT->"Format";caseCONFLICT->"Taken";caseERROR->"Error";caseSAVED->"Saved";};varbadge=newSpan(lbl);vartheme=switch(row.getStatus()){caseVALID,SAVED->"badge success";caseCONFLICT,INVALID_FORMAT,ERROR->"badge error";default->"badge";};badge.getElement().getThemeList().add(theme);if(row.getMsg()!=null&&!row.getMsg().isBlank())badge.setTitle(row.getMsg());returnbadge;}).setHeader("Status").setAutoWidth(true);grid.addComponentColumn(row->{vardel=newButton("✕",e->{varitems=newArrayList<>(grid.getListDataView().getItems().toList());items.remove(row);grid.setItems(items);});del.getElement().setProperty("title","Remove");returndel;}).setHeader("").setAutoWidth(true);grid.setItems(newArrayList<>());grid.setAllRowsVisible(false);grid.setHeight("320px");}

Here, it is clearly visible how all relevant information – alias, preview URL, status, and deletion action – converges in a separate, self-contained table. The status display plays a central role in providing the user with feedback on format errors, conflicts, successful validation or entries that have already been saved.

The real intelligence of the editor lies in theparseBulk() andvalidateRow() logic.parseBulk() takes over the task of processing the free text input, detecting duplicates and creating new lines in the grid:

xmlCopy

private void parseBulk() { var text = Objects.requireNonNullElse(bulk.getValue(), ""); var tokens = Arrays.stream(text.split("[,;\\s]+")) .map(String::trim) .filter(s -> !s.isBlank()) .distinct() .toList(); if (tokens.isEmpty()) { Notification.show("No aliases to insert", 2000, Notification.Position.TOP_CENTER); return; } Set<String> existing = grid.getListDataView().getItems() .map(Row::getAlias) .filter(Objects::nonNull) .collect(Collectors.toCollection(() -> new TreeSet<>(String.CASE_INSENSITIVE_ORDER))); var view = grid.getListDataView(); int added = 0; for (String tok : tokens) { if (existing.contains(tok)) continue; var r = new Row(tok); validateRow(r); view.addItem(r); existing.add(tok); added++; } grid.getDataProvider().refreshAll(); bulk.clear(); Notification.show("Inserted: " + added, 2000, Notification.Position.TOP_CENTER);}

The validation of individual rows is encapsulated invalidateRow() and follows a clear step-by-step model of format checking, duplicate checking, and optional server querying:

javaCopy

privatevoidvalidateRow(Rowr){vara=Objects.requireNonNullElse(r.getAlias(),"");if(!a.matches(RX)){r.setStatus(Status.INVALID_FORMAT);r.setMsg("3–64: A–Z a–z 0–9 - _");return;}longsame=grid.getListDataView().getItems().filter(x->x!=r&&Objects.equals(a,x.getAlias())).count();if(same>0){r.setStatus(Status.CONFLICT);r.setMsg("Duplicate in list");return;}if(isAliasFree!=null){try{if(!isAliasFree.apply(a)){r.setStatus(Status.CONFLICT);r.setMsg("Alias taken");return;}}catch(Exceptionex){r.setStatus(Status.ERROR);r.setMsg("Check failed");return;}}r.setStatus(Status.VALID);r.setMsg("");}

Together with thestatus enum and the innerrow class, this results in a self-contained, easily comprehensible state machine for each alias:

javaCopy

publicenumStatus{NEW,VALID,INVALID_FORMAT,CONFLICT,ERROR,SAVED}publicstaticfinalclassRow{privatestringalias;privatestatusstatus=Status.NEW;privateStringmsg="";Row(Stringa){this.alias=a;}publicStringgetAlias(){returnalias;}publicvoidsetAlias(Stringa){this.alias=a;}publicStatusgetStatus(){returnstatus;}publicvoidsetStatus(Statuss){this.status=s;}publicStringgetMsg(){returnmsg;}publicvoidsetMsg(Stringm){this.msg=m;}}

Overall, the revision of the MultiAliasEditorStrict shows how even smaller UI components can benefit significantly from a clean structure, clear responsibilities and consistent behaviour. The component is now more stable, more comprehensible and easier to integrate – ideal conditions for future extensions or adjustments.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-extracting-components-part-1/Sun, 21 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-extracting-components-part-1/Today marks a crucial step in the evolution of the URL shortener’s user interface. After the focus in the past few days was mainly on functional enhancements – from filter and search functions to bulk operations – this day is dedicated to a structural fine-tuning: the refactoring of central UI components. This refactoring not only serves to clean up the code but also creates a clear, modular basis for future extensions as well as a significantly improved developer experience.<![CDATA[

Today marks a crucial step in the evolution of the URL shortener’s user interface. After the focus in the past few days was mainly on functional enhancements – from filter and search functions to bulk operations – this day is dedicated to a structural fine-tuning: the refactoring of central UI components. This refactoring not only serves to clean up the code but also creates a clear, modular basis for future extensions as well as a significantly improved developer experience.

The source code for this article can be found on GitHub at:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-11

Here is a screenshot of the current development state from the user’s perspective.

The focus is on dividing the previously monolithic OverviewView into clearly defined, reusable components such as the BulkActionsBar and the SearchBar. This separation improves maintainability, increases clarity and makes it easier to test individual functional areas. At the same time, this will pave the way for further UI optimisations and interaction patterns to be implemented gradually in the coming days.

Motivation for refactoring

As an application’s functionality increases, so does its source code complexity. Especially in a UI that is continuously expanded and adapted to new requirements, this quickly leads to monolithic structures in which presentation, logic and state management are closely interwoven. This is precisely the situation that had developed in the OverviewView over the past few days of development: a central view that had to take on more and more tasks and thus became increasingly difficult to understand, extend, and test.

Refactoring today, therefore, starts at a fundamental point. The goal is not to introduce new features, but to create a clear, modular foundation that makes future expansions much easier. Individual UI building blocks are removed from the overloaded view and formed into independent components, with responsibility-specific tasks, clear communication channels, and significantly improved reusability. This not only reduces the complexity within the view itself, but also makes the entire architecture more robust to change.

At the same time, this refactoring strengthens the developer experience: The structure of the code becomes more comprehensible, components can be improved in isolation, and future changes – such as to the search logic or bulk operations – can be made in the right place in a targeted manner. The result is a UI design that is more stable, more flexible and easier to maintain in the long term.

Why is this step a turning point today

Today marks a critical moment in the development of the URL shortener’s user interface, as the focus shifts for the first time from new features to the code’s structural quality. While the previous steps were primarily aimed at delivering visible improvements for the user, it is now a matter of ensuring the foundation on which these functions remain solid and scalable over the long term. Such refactoring creates clarity in areas that had previously grown instead of planned – and that’s precisely what makes this day a turning point.

By separating the central UI building blocks, an architecture emerges that no longer consists of a single, heavyweight view but of clearly defined components that perform precisely defined tasks. This structural realignment opens up new possibilities for expansion, halts progress on technical debt reduction, and ensures that upcoming features no longer have to be integrated into a confusing block. Instead, each feature can be implemented in the right place without destabilising existing areas.

This step thus makes a decisive contribution to long-term development: reduced friction in the code, fewer side effects, and greater clarity. So today it’s not about visible innovations, but about the quality of the foundation – and that’s exactly what makes working on upcoming expansions much more efficient, stable and enjoyable.

Overview of the most important changes

Today, the focus is on replacing the previously centralOverviewView, which had increasingly assumed responsibility over the course of development and had become correspondingly confusing. By extracting independent components, this complexity is decomposed into clearly defined building blocks.

An essential part of this restructuring is the introduction of the newBulkActionsBar, which bundles all mass operations, making both the code and the user interface more straightforward. The newSearchBar also creates a dedicated area that includes search and filter functions, making it easier to expand. Both components not only simplify the view but also provide a foundation for consistent design and recurring interaction patterns throughout the user interface.

In addition, the internal logic of the overview has been streamlined: state management, event handling, and grid interactions have been rearranged to reduce unwanted dependencies and enable targeted future changes. Minor improvements, such as the unified formatting of specific UI components, also contribute to the overall impression of a clean, structured codebase.

Why are UI components removed from views?

In mature user interfaces, a close interlocking of layout, interaction logic and state management often arises. As a result, central views become increasingly extensive over time and lose their original character. A clear entry point of the application becomes a complex block whose internal logic is difficult to understand. This is exactly where the separation of independent components comes in: It serves to clearly separate responsibilities and shift complexity to where it belongs – into small, clearly defined building blocks.

By extracting UI components, the view is reduced to its core task: orchestrating the interaction of several well-defined elements. Each component takes on a clearly defined role – be it managing search filters, triggering bulk operations, or displaying individual UI sections. This modular approach improves readability, fosters a more natural understanding of the architecture, and significantly reduces side effects during refactoring.

Another advantage is reusability. Components that are only loosely coupled to the overall view can be used flexibly elsewhere, expanded, or improved in isolation. This not only creates a robust structure but also enables more sustainable development practices, allowing new functions to be implemented without major interventions in existing areas. The separation of UI components is, therefore, an essential step in keeping a growing project stable and clear in the long term.

The new BulkActionsBar

Bulk operations are a function that plays an increasingly important role in the daily use of a URL shortener. As the number of entries grows, the need to edit several items at the same time increases – for example, to deactivate expired links, delete outdated campaigns or manage a larger number of new entries together. Such operations are not only a convenience feature but also an essential part of efficient workflows.

In many projects, bulk operations are first integrated directly into the main view. This works well at first, but in the long run, it causes view overload. Buttons, health checks, and more complex interaction logics begin to blend between Grid and View. The result is a structure that is difficult to maintain and in which extensions always carry the risk of unwanted side effects.

The decision to outsource bulk operations to a separate component creates a clear separation: theBulkActionsBar takes over the entire UI-related part of mass actions and thus forms a dedicated functional area. It bundles all relevant actions in one place, ensures a consistent user experience, and reduces complexity in the higher-level view. This clear delineation makes it much easier to add new actions, extend existing ones, or adapt the presentation to future design requirements.

In this way, outsourcing bulk operations makes a decisive contribution to a more stable and flexible architecture – and ensures that the user interface remains clear and intuitive even as requirements grow.

Structure of the BulkActionsBar component

The BulkActionsBar is an independent UI component that bundles all mass actions and presents them in a clearly structured way. Their structure follows the principle that an element should cover exactly one area of responsibility: In this case, the initiation of collection actions without itself containing logic for data processing. This keeps it easy to understand and flexible to use.

The central component of the BulkActionsBar is a clearly defined collection of interaction elements – typically buttons or icons – each of which triggers a specific action. These elements are compactly arranged in a horizontal layout, making them highly visible and intuitively accessible in the user interface. This is complemented by a mechanism that controls the visibility and activability of actions based on whether and how many entries are selected in the grid. In this way, the component not only remains clear but also guides the user through clear interaction instructions.

Another essential aspect of the structure is the abstraction of event processing. The component itself does not perform any operations; instead, it signals to the higher-level view via events or callback functions which action to trigger. This creates loose coupling between the UI and the logic, which both facilitates testing and allows adjustments to individual areas without unintentionally affecting other components.

A practical example of this is the basic structure of the class that defines the BulkActionsBar:

xmlCopy

public class BulkActionsBar extends Composite<HorizontalLayout> implements HasLogger { private final URLShortenerClient urlShortenerClient; private final Grid<ShortUrlMapping> grid; private final OverviewView holdingComponent; private final Button bulkDeleteBtn = new Button(new Icon(VaadinIcon.TRASH)); private final Button bulkSetExpiryBtn = new Button(new Icon(VaadinIcon.CLOCK)); private final Button bulkClearExpiryBtn = new Button(new Icon(VaadinIcon.CLOSE_CIRCLE)); private final Button bulkActivateBtn = new Button(new Icon(VaadinIcon.PLAY)); private final Button bulkDeactivateBtn = new Button(new Icon(VaadinIcon.STOP)); private final Span selectionInfo = new Span(); public BulkActionsBar(URLShortenerClient urlShortenerClient, Grid<ShortUrlMapping> grid, OverviewView holdingComponent) { this.urlShortenerClient = urlShortenerClient; this.grid = grid; this.holdingComponent = holdingComponent; buildBulkBar(); addListeners(); }}

Here, it becomes clear how the component encapsulates all elements relevant to mass actions: It knows theURLShortenerClient, the underlyingGrid<ShortUrlMapping>, and the higher-levelOverviewView, yet remains clearly delimited as an independent component. All buttons and the selection information area are defined within the class and are initialised in the constructor.

The actual visual structure of the BulkActionsBar is encapsulated in its own method:

javaCopy

privatevoidbuildBulkBar(){---Commonstyle---bulkBar().getStyle().set("background","var(--lumo-contrast-5pct)").set("padding","0.4rem 0.8rem").set("border-radius","var(--lumo-border-radius-m)").set("border-bottom","1px solid var(--lumo-contrast-20pct)");---buttonsetup---setupIconButton(bulkDeleteBtn,VaadinIcon.TRASH,"Delete selected links","var(--lumo-error-color)");setupIconButton(bulkSetExpiryBtn,VaadinIcon.CALENDAR_CLOCK,"Set expiry for selected","var(--lumo-primary-color)");setupIconButton(bulkClearExpiryBtn,VaadinIcon.CALENDAR_CLOCK,"Clear expiry for selected","var(--lumo-secondary-text-color)");setupIconButton(bulkActivateBtn,VaadinIcon.CHECK_CIRCLE,"Activate selected","var(--lumo-success-color)");setupIconButton(bulkDeactivateBtn,VaadinIcon.CLOSE_CIRCLE,"Deactivate selected","var(--lumo-error-color)");bulkBar().removeAll();selectionInfo.getStyle().set("opacity","0.7");selectionInfo.getStyle().set("font-size","var(--lumo-font-size-s)");selectionInfo.getStyle().set("margin-right","var(--lumo-space-m)");bulkBar().add(selectionInfo,bulkDeleteBtn,bulkSetExpiryBtn,bulkClearExpiryBtn,bulkActivateBtn,bulkDeactivateBtn);bulkBar().setWidthFull();bulkBar().setSpacing(true);bulkBar().setDefaultVerticalComponentAlignment(FlexComponent.Alignment.CENTER);bulkBar().setVisible(false);}

The method shows how style, layout and interaction elements are brought together in a central place. The BulkActionsBar thus defines its own visual and functional cluster within the interface, while the URLShortenerClient and the OverviewView still handle the actual execution of the actions.

Overall, this setup enables a clean separation of display and behaviour, provides a robust foundation for future expansion, and integrates seamlessly with the application’s modular architecture.

Using the BulkActionsBar in the OverviewView

The BulkActionsBar is only valuable for interaction with the OverviewView, because there it becomes clear how much outsourcing mass actions simplifies the user interface structure. While these actions were previously implemented directly in the view itself, mixing a large number of event handlers, UI elements, and logic, the BulkActionsBar now handles this entire functional area. The OverviewView only needs to define when the bar becomes visible, which entries are selected, and how to respond to the triggered actions.

The integration follows a clear pattern: As soon as the user selects one or more rows in the grid, the view displays the BulkActionsBar and passes it the current selection state. The component itself does not perform any operations, but signals which action has been triggered via clearly defined methods and events. This creates loose coupling, which increases clarity and significantly improves visibility.

Another advantage is evident when handling status changes. The OverviewView no longer has to activate or deactivate buttons individually or control complex UI dependencies. Instead, a single central feedback to the BulkActionsBar is sufficient; it updates itself. This interaction not only makes the code leaner, but also prevents errors that could easily occur with multiple scattered logics.

A central entry point for the integration is to initialise the BulkActionsBar directly as a field of the OverviewView:

javaCopy

privatefinalBulkActionsBarbulkBar=newBulkActionsBar(urlShortenerClient,grid,this);

This provides the view with a fully configured instance that knows theURLShortenerClient, the grid, and the view itself. The component is then integrated into the constructor:

javaCopy

add(bulkBar);add(grid);

Significant is the interaction with the grid’s selection listener. Here, the view decides whether the BulkActionsBar is visible and what information it should display:

kotlinCopy

grid.addSelectionListener(event->{varall=event.getAllSelectedItems();booleanhasSelection=!all.isEmpty();bulkBar.setVisible(hasSelection);if(hasSelection){intcount=all.size();Stringlabel=count==1?"link selected":"links selected";bulkBar.selectionInfoText(count+" "+label+" on page "+currentPage);}else{bulkBar.selectionInfoText("");}bulkBar.setButtonsEnabled(hasSelection);});

This section clarifies the precise distribution of roles: the view recognises the current selection state, updates the visibility of the BulkActionsBar, and forwards relevant status information. The component itself remains completely free of logic for selection management.

The triggering of the actual actions also remains bundled in the view. An example of this is deleting using keyboard shortcuts:

sqlCopy

current.addShortcutListener(_->{if(!grid.getSelectedItems().isEmpty()){bulkBar.confirmBulkDeleteSelected();}},Key.DELETE);

This once again shows the interplay of clearly defined responsibilities: the BulkActionsBar encapsulates the action UI, and the OverviewView determines the context in which it is executed.

Overall, the use of the BulkActionsBar in the OverviewView clearly demonstrates how a clear separation of responsibilities improves the architecture of a Vaadin application. The View refocuses on its core task – presenting and updating the data – while the component encapsulates and consistently delivers all the interaction around mass actions. The BulkActionsBar in the OverviewView demonstrates how a clear separation of responsibilities improves the architecture of a Vaadin application. The View refocuses on its core task – presenting and updating the data – while the component encapsulates and consistently delivers all the interaction around mass actions.

Code Comparison: Before vs. After

The difference between the previous implementation of bulk operations and the new, component-based structure is immediately apparent. Whereas previously all elements, buttons and dialogues were anchored directly in the OverviewView, distributed over numerous event handlers and UI areas, the code is now much more modular. The BulkActionsBar is a standalone component that encapsulates the entire UI for bulk actions. As a result, the OverviewView not only had to accommodate less logic but also gained overall clarity.

The difference is evident in the reduction of responsibilities within the view. Before the refactoring, the OverviewView handled all aspects: providing the buttons, displaying the bar, handling selection, opening and executing dialogues, and providing user feedback. This resulted in an extensive, tightly coupled block of code that was difficult to maintain and error-prone.

After the refactoring, the structure has improved significantly. All UI-related logic of bulk operations is now exclusively in the BulkActionsBar. The OverviewView is limited to recognising the selection and forwarding the necessary information. The component itself still triggers the actions, but the view is no longer involved in the UI’s visual or structural layout. This change improves readability, as each functional area is now located where it belongs. In addition, the risk of unintended side effects is significantly reduced, as changes to the BulkActionsBar no longer require direct intervention in the OverviewView. To clarify the difference, it is worth reviewing typical places that were previously located directly in the OverviewView and are now outsourced. A classic example is the handling of the bulk deletion operation. In the past, the OverviewView itself handled dialogue, loops, error handling, and refresh. Today, the entire logic can be found clearly closed in the BulkActionsBar:

New structure – outsourced bulk delete logic:

javaCopy

publicvoidconfirmBulkDeleteSelected(){varselected=grid.getSelectedItems();if(selected.isEmpty()){Notifications.noSelection();return;}Dialogdialog=newDialog();dialog.setHeaderTitle("Delete "+selected.size()+" short links?");varexampleCodes=selected.stream().map(ShortUrlMapping::shortCode).sorted().limit(5).toList();if(!exampleCodes.isEmpty()){Stringpreview=String.join(", ",exampleCodes);if(selected.size()>5)preview+=", ...";dialog.add(newText("Examples: "+preview));}else{dialog.add(newText("Delete selected short links?"));}Buttonconfirm=newButton("Delete",_->{intsuccess=0;intfailed=0;for(varm:selected){try{booleanok=urlShortenerClient.delete(m.shortCode());if(ok)success++;elsefailed++;}catch(IOExceptionex){logger().error("Bulk delete failed for {}",m.shortCode(),ex);failed++;}}dialog.close();grid.deselectAll();holdingComponent.safeRefresh();Notifications.deletedAndNotDeleted(success,failed);});confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY,LUMO_ERROR);Buttoncancel=newButton("Cancel",_->dialog.close());dialog.getFooter().add(newHorizontalLayout(confirm,cancel));dialog.open();}

Here you can see that the entire interaction – from the UI to the error handling to the update – now takes placewithin the component and no longer burdens the OverviewView.

New structure – View only signals the selection:

kotlinCopy

grid.addSelectionListener(event->{varall=event.getAllSelectedItems();booleanhasSelection=!all.isEmpty();bulkBar.setVisible(hasSelection);if(hasSelection){intcount=all.size();Stringlabel=count==1?"link selected":"links selected";bulkBar.selectionInfoText(count+" "+label+" on page "+currentPage);}else{bulkBar.selectionInfoText("");}bulkBar.setButtonsEnabled(hasSelection);});

This shows the new role distribution: the OverviewView only controls visibility and status display. In the past, the entire operation logic would have been implemented here as well. Another example is opening the bulk set expiry dialogues. This logic is now also completely in the component and no longer distributed across the View:

javaCopy

bulkSetExpiryBtn.addClickListener(_->openBulkSetExpiryDialog());

The entire dialogue logic is then located in:

javaCopy

privatevoidopenBulkSetExpiryDialog(){...}

Result:
The clear separation of responsibilities results in:

• significantly fewer lines of code in the OverviewView,
• a more structured, modular architecture,
• less coupling,
• improved maintainability and expandability.

Overall, this results in a more maintenance-friendly, extensible and clear structured architecture.

Search requirements

A practical search function is a central part of any administrative interface, especially when the number of entries is continuously growing. In the context of URL shorteners, this means that users can quickly and specifically access specific short links – regardless of whether they are looking for a particular shortcode, a URL fragment, the active status or time criteria. The previous implementation offered only simple filter options and was tightly coupled to the OverviewView, making both extensibility and maintenance difficult.

However, the requirements for a modern search component go far beyond a simple text field. It must support multiple filter criteria, respond flexibly to new fields, and adapt dynamically to the backend data structure. At the same time, it should offer the user an intuitive, consistent and clutter-free interface. This includes clear input fields, understandable labels, sensible default values, and the ability to change or reset search parameters quickly.

Technical reliability is just as crucial as user-friendliness: the search function must not burden the backend unnecessarily, support high-performance queries, and use server-side filters efficiently. A clean separation between UI, filter logic and data retrieval not only enables a better overview, but also later extensions – such as additional filter fields, sorting options or advanced functions such as combining several criteria.

The introduction of the new SearchBar addresses these requirements. It serves as the comprehensive filter and control centre for the overview and ensures that the view itself is decoupled from the filter logic. In doing so, it lays the foundation for a scalable and user-friendly search and filtering experience.

Structure and internal logic of the SearchBar

The SearchBar is designed as an independent UI component that aggregates all filter and search parameters from the OverviewView and presents them in a clearly structured format. This component aims to centralise previously scattered filter logic and significantly improve view clarity. Whereas previously individual input fields and sorting parameters were defined directly in the OverviewView, the SearchBar now assumes full responsibility for their management.

Its structure follows a modular concept: Each input – be it a text filter, a sorting criterion, the number of displayed elements or filtering by properties such as active status or expiration date – is clearly demarcated within the component and is processed independently. This not only ensures better logical separation but also enables flexible expansion with additional filters without requiring adjustments to the view itself.

The internal logic of the SearchBar works closely with the backend. Based on the user-selected parameters, the component creates structured filter objects that can be passed to the server consistently. Instead of collecting individual parameters loosely or combining them in the view, they are merged in a clearly defined process: validated, normalised, and then transferred to a backend request.

The central method buildFilter, which creates aUrlMappingListRequest objectfrom the UI inputs,shows what this process looks like in concrete terms:

kotlinCopy

publicUrlMappingListRequestbuildFilter(Integerpage,Integersize){UrlMappingListRequest.Builderb=UrlMappingListRequest.builder();ActiveStateactiveStateValue=activeState.getValue();logger().info("buildFilter - activeState == {}",activeStateValue);if(activeStateValue!=null&&activeStateValue.isSet()){b.active(activeStateValue.toBoolean());}if(codePart.getValue()!=null&&!codePart.getValue().isBlank()){b.codePart(codePart.getValue());}if(urlPart.getValue()!=null&&!urlPart.getValue().isBlank()){b.urlPart(urlPart.getValue());}if(fromDate.getValue()!=null&&fromTime.getValue()!=null){varzdt=ZonedDateTime.of(fromDate.getValue(),fromTime.getValue(),ZoneId.systemDefault());b.from(zdt.toInstant());}elseif(fromDate.getValue()!=null){varzdt=fromDate.getValue().atStartOfDay(ZoneId.systemDefault());b.from(zdt.toInstant());}if(toDate.getValue()!=null&&toTime.getValue()!=null){varzdt=ZonedDateTime.of(toDate.getValue(),toTime.getValue(),ZoneId.systemDefault());b.to(zdt.toInstant());}elseif(toDate.getValue()!=null){varzdt=toDate.getValue().atTime(23,59).atZone(ZoneId.systemDefault());b.to(zdt.toInstant());}if(sortBy.getValue()!=null&&!sortBy.getValue().isBlank())b.sort(sortBy.getValue());if(dir.getValue()!=null&&!dir.getValue().isBlank())b.dir(dir.getValue());if(page!=null&&size!=null){b.page(page).size(size);}varfilter=b.build();logger().info("buildFilter - {}",filter);returnfilter;}

The code shows how all relevant filter fields – active status, shortcode and URL parts, time windows, and sorting and paging information – are consolidated in a single place. The SearchBar thus serves as a translator between UI inputs and the domain-specific filter structure in the backend. This creates a robust interface that both reduces errors and facilitates future expansions. At the same time, the SearchBar ensures a consistent user experience. Changes to a field automatically update the results list, while sensible default values and a consistent display ensure a familiar user experience. This combination of structural clarity, technical precision, and ease of use makes the SearchBar a central building block of the project’s modern UI architecture.

Improvements compared to the previous solution

From a technical standpoint, the new solution offers greater robustness. The processing is now handled entirely in the SearchBar, using structured data objects that are transmitted directly to the backend. This minimises errors caused by inconsistent or incomplete filter parameters. At the same time, the clear structure makes it easy to add new filters without jeopardising existing functionality.

A key example of the improvements is how changes to the SearchBar filter are handled. Whereas previously the OverviewView had to check and react to each value, the SearchBar now handles this task independently. This is already evident in the ValueChange listeners of the individual input fields:

javaCopy

codePart.addValueChangeListener(_->holdingComponent.safeRefresh());urlPart.addValueChangeListener(_->holdingComponent.safeRefresh());activeState.addValueChangeListener(_->{holdingComponent.setCurrentPage(1);holdingComponent.safeRefresh();});pageSize.addValueChangeListener(e->{holdingComponent.setCurrentPage(1);holdingComponent.setGridPageSize(e.getValue());holdingComponent.safeRefresh();});

These listeners make it clear that the SearchBar takes complete control of the filters’ behaviour and automatically updates the view. Previously, this logic was distributed in the OverviewView. Global search has also been significantly improved. It used to be a standalone input field with no real integration, but now it dynamically controls the specific search fields and ensures consistent filters:

kotlinCopy

globalSearch.addValueChangeListener(e->{varv=Optional.ofNullable(e.getValue()).orElse("");if(searchScope.getValue().equals("Shortcode")){codePart.setValue(v);urlPart.clear();}else{urlPart.setValue(v);codePart.clear();}});

As a result, global search has become a real entry point for filter logic, rather than being an additional field with no straightforward integration.

Another big step forward is the introduction of the reset mechanism, which resets all filters in a targeted manner while ensuring that the UI returns to a consistent state:

javaCopy

resetBtn.addClickListener(_->{try(var_=withRefreshGuard(true)){resetElements();holdingComponent.setCurrentPage(1);}catch(Exceptione){thrownewRuntimeException(e);}});

Finally, switching between simple and advanced search also shows the new structural quality of the SearchBar:

javaCopy

advanced.addOpenedChangeListener(ev->{booleannowClosed=!ev.isOpened();if(nowClosed){applyAdvancedToSimpleAndReset();}else{setSimpleSearchEnabled(false);}});

Overall, the new SearchBar is far superior in both functionality and architecture. Their listeners, control logic, and consistent handling of filter elements form the basis for a modern, scalable, and maintenance-friendly search architecture. It is not only functionally better, but also forms the basis for a modern, scalable and maintenance-friendly search architecture.

Interaction of the SearchBar with grid and backend

The SearchBar only shows its full strength when interacting with the grid and the backend that powers it. While it serves as a central input component for all search and filter parameters on the interface, it technically forms the link between user interaction and server-side data supply. It is precisely in this role that it becomes clear how crucial the decoupling of the filter logic from the actual view is for performance, maintainability and extensibility.

In the first step, the SearchBar accepts all user input – from global search texts to specific shortcode or URL parts to date ranges, sort fields and the active status. These values are no longer processed ad hoc in the OverviewView, but collected, validated and harmonised in a structured form within the SearchBar. This keeps filter states consistent and traceable, even when multiple input fields are changed simultaneously.

The second step is the interaction with the grid. As soon as a relevant filter value changes, the SearchBar notifies the OverviewView of the updated state. This, in turn, triggers a grid update without requiring the individual filter criteria to be known or processed. The grid then calls the backend via its DataProvider and receives a filtered subset of the data based on the SearchBar’s requirements. This creates a clearly separated yet closely interlinked system of input, data retrieval and presentation.

On the backend, the advantage of a structured filter is particularly evident in how the OverviewView configures its DataProvider. The SearchBar always provides a consistent filter object that is passed directly to the backend calls. Central to this is the initialisation of the DataProvider in the OverviewView:

xmlCopy

private void initDataProvider() { dataProvider = new CallbackDataProvider<>( q -> { final int uiSize = Optional.ofNullable(searchBar.getPageSize()).orElse(25); final int pageStart = (currentPage - 1) * uiSize; final int vLimit = q.getLimit(); final int vOffset = q.getOffset(); final int effectiveLimit = (vLimit > 0) ? vLimit: uiSize; final int effectiveOffset = pageStart + vOffset; final int page = (effectiveLimit > 0) ? (effectiveOffset / effectiveLimit) + 1 : 1; final int size = (effectiveLimit > 0) ? effectiveLimit: uiSize; final UrlMappingListRequest req = searchBar.buildFilter(page, size); try { final List<ShortUrlMapping> items = urlShortenerClient.list(req); return items.stream(); } catch (IOException ex) { logger().error("Error fetching (page={}, size={})", page, size, ex); Notifications.loadingFailed(); return Stream.empty(); } }, _ -> { try { final UrlMappingListRequest base = searchBar.buildFilter(null, null); totalCount = urlShortenerClient.listCount(base); final int uiSize = Optional.ofNullable(searchBar.getPageSize()).orElse(25); final int pageStart = (currentPage - 1) * uiSize; final int remaining = Math.max(0, totalCount - pageStart); final int pageCount = Math.min(uiSize, remaining); refreshPageInfo(); return pageCount; } catch (IOException ex) { logger().error("Error counting", ex); totalCount = 0; refreshPageInfo(); return 0; } } ); grid.setPageSize(Optional.ofNullable(searchBar.getPageSize()).orElse(25)); grid.setDataProvider(dataProvider);}

This makes it clear how closely the grid, SearchBar, and backend interact: the DataProvider calculates the effective query parameters from the current paging information and page size, then builds aUrlMappingListRequest from the SearchBar and passes it directly to theURLShortenerClient. The SearchBar is also used for the counting function, this time without paging parameters, to determine the total number of entries.

The advantage of this structure is that the OverviewView itself does not need to know the details of the filter fields. It delegates the entire filter configuration to the SearchBar and focuses solely on controlling the grid and paging. Changes to the filters – such as additional criteria or changed default values – can be made entirely in the SearchBar without having to adjust the DataProvider or the View.

Overall, the interaction among SearchBar, Grid, and the backend shows a cleanly orchestrated data-flow model: Users change a filter, the SearchBar generates a unique search query, the DataProvider requests the appropriate data via theURLShortenerClient, and the Grid presents the results. This consistent, clearly structured process makes the entire interface much more stable, understandable and responsive.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-de-activate-mappings-part-2/Sat, 20 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-de-activate-mappings-part-2/What has happened so far? In the first part of this article, the new active/inactive model for shortlinks was introduced and anchored at the architectural level. Based on the technical rationale, it was shown that a pure expiration date is insufficient for modern use cases and that an explicit activity status is required.<![CDATA[

What has happened so far?

In the first part of this article, the new active/inactive model for shortlinks was introduced and anchored at the architectural level. Based on the technical rationale, it was shown that a pure expiration date is insufficient for modern use cases and that an explicit activity status is required.

Based on this, the technical foundations were laid: the core domain model was extended with anactiveflag, the DTOs were adapted accordingly, and the serialisation was designed to ensure backward compatibility. In addition, the administrative REST endpoints have been expanded to enable targeted setting, querying, and filtering of activity status. The redirect behaviour has also been clarified so that deactivated and expired shortlinks can be distinguished by clearly defined HTTP status codes.

The structural basis of the active/inactive model is thus fully established. In the second part, the focus is on practical use: how the Java client maps these new capabilities, how users can conveniently control activity status via the Vaadin interface, and how this results in consistent, efficient workflows in everyday use.

The source code for this project‘s status can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-10

Java Client Enhancements

After the server API had been extended to include functions for activating and deactivating shortlinks, the Java client also had to be adapted accordingly. After all, it is the primary interface for many users to work programmatically with the URL shortener – whether in the context of desktop tools, automations, CI/CD pipelines or embedded systems.

Chapter 5 details how new capabilities have been added to the client to support the active/inactive model fully. These include:

• the targeted switching of the activity status,
• the initial setting of the activity and expiration date when creating a shortlink,
• as well as editing existing mappings, including the new activity field.

The extensions are based on a consistent design approach: simple parameters, clear method focus, strict validation, and traceable error handling. For users, this creates an API that is not only complete, but also intuitive to use – regardless of whether individual values are changed or complete mappings are rebuilt.

New API:toggleActive(shortCode, active)

To allow the user to control the activity status of a shortlink not only via the REST API, but also conveniently via the Java client, a new method has been added to the client API. This method is the functional equivalent of the toggle endpoint on the server side and allows shortlinks to be enabled or disabled directly from applications, scripts, or automations.

The new API methodtoggleActive(shortCode, active) does exactly this. It ensures that all relevant information is transmitted to the server in the correct structure and that the server’s response is converted into a suitable representation. With a clear focus on switching activity status, the user eliminates the need to build full update objects or send unnecessary data.

Another advantage of this method is its simplicity: the user only needs to specify the shortcode of the link to be changed as well as the desired new status. The client’s internal logic takes care of everything else – from creating the appropriate request payload to interpreting the server response. This makes it particularly intuitive to use and reduces potential sources of error.

In the next step, we will look at the concrete implementation of this method using the source code. The following implementation comes directly from theURLShortenerClient:

javaCopy

publicbooleantoggleActive(StringshortCode,booleanactive)throwsIOException{logger().info("Toggle Active shortCode='{}' active='{}'",shortCode,active);if(shortCode==null||shortCode.isBlank()){thrownewIllegalArgumentException("shortCode must not be null/blank");}finalURIuri=serverBaseAdmin.resolve(PATH_ADMIN_TOGGLE_ACTIVE+"/"+shortCode);finalURLurl=uri.toURL();logger().info("Toggle Active - {}",url);finalHttpURLConnectioncon=(HttpURLConnection)url.openConnection();con.setRequestMethod("PUT");con.setDoOutput(true);con.setRequestProperty(CONTENT_TYPE,JSON_CONTENT_TYPE);con.setRequestProperty(ACCEPT,APPLICATION_JSON);con.setConnectTimeout(CONNECT_TIMEOUT);con.setReadTimeout(READ_TIMEOUT);varreq=newToggleActiveRequest(shortCode,active);finalStringbody=toJson(req);logger().info("Toggle Active - request body - '{}'",body);try(OutputStreamos=con.getOutputStream()){os.write(body.getBytes(UTF_8));}finalintcode=con.getResponseCode();logger().info("Toggle Active - responseCode {}",code);if(code==200||code==204||code==201){drainQuietly(con.getInputStream());returntrue;}if(code==404){logger().info("shortCode not found.. {}",shortCode);drainQuietly(con.getErrorStream());returnfalse;}finalStringerr=readAllAsString(con.getErrorStream());thrownewIOException("Unexpected response: "+code+", body="+err);}

The method starts with basic validation and logging. TheshortCode parameter must be set because it uniquely identifies the shortlink to change. If the value is null or empty, the client throws anIllegalArgumentException even before an HTTP call is made.

The next step is to create the destination URL for the API call. The server’s administrative base path (serverBaseAdmin) is combined with the toggle endpoint path segment. Thanks to this dynamic composition, the client remains flexible in different deployment environments.

The client then opens an HTTP connection and configures it for a PUT request. The method sets the expected header fields, includingContent-Type (for JSON) andAccept, to define the expected response type.setDoOutput(true) indicates that a request body is included in the request.

For the actual payload, an instance ofToggleActiveRequest is created, whichconsists of a shortCode and the desired newactive state. This structure is serialized usingtoJson and then written to the output stream of the connection.

After the request is sent, the method reads the HTTP status code viacon.getResponseCode(). The implementation distinguishes between three main cases:

1. Successful state change (200, 204, or 201): The method flushes the InputStream viadrainQuietly and returnstrue. This signals to the user that the shortlink has been updated successfully.
2. Shortlink not found (404): Again, the ErrorStream is emptied. However, the method returnsfalse to make it clear to the user that the shortlink does not exist and therefore cannot be updated.
3. All other error cases : In the event of unexpected or erroneous responses, the ErrorStream is read and packaged together with the HTTP status code in anIOException. This forces the calling code to handle unforeseen errors and prevents such states from being silently ignored.

Thus,toggleActive provides a clearly defined, robust API for switching activity status via the Java client. It follows the same design principles as the client’s other methods: clear validation, consistent error handling, meaningful logging, and lean, JSON-based communication. The implementation thus integrates seamlessly into the existing client architecture and provides a simple yet effective extension of functionality.

AdvancedcreateCustomMapping(...)

In addition to the possibility to activate or deactivate existing shortlinks afterwards, the process for creating new shortlinks has also been expanded. To allow users to determine whether a shortlink is active when making it, thecreateCustomMapping(...) method is used in the Java client.

Before this adjustment, a shortlink could be created using only its fundamental properties – shortcode, original URL, and optional expiration date. The activity status was set implicitly or could only be controlled via later processing steps. With the new extension, the user can determine during creation whether a shortlink is active or inactive.

The method follows the same principles as the client’s other functions: it focuses on clear, simple, and reliable communication with the server. The user only provides the required input values. At the same time, the client takes care of the entire technical processing of the request – from creating the request object to JSON serialisation and interpreting the server response.

This expansion allows new use cases to be realised. For example, a shortlink can already be created, but only activated later – for example, synchronously with a release time, a marketing campaign or automatically in CI/CD pipelines. At the same time, the uniform data structure ensures that the activity status of a shortlink is treated consistently both when creating and editing.

In the next step, we examine the concrete implementation of this method using the relevant source code sections and analyse how the extended parameters are integrated into the creation process.

The extension appears in theURLShortenerClient in the form of two overloaded methods: a simple variant and an extended version with expiration and activity parameters:

javaCopy

publicShortUrlMappingcreateCustomMapping(Stringalias,Stringurl)throwsIOException{logger().info("Create custom mapping alias='{}' url='{}'",alias,url);returncreateCustomMapping(alias,url,null,null);}publicShortUrlMappingcreateCustomMapping(Stringalias,Stringurl,InstantexpiredAtOrNull,BooleanactiveOrNull)throwsIOException{logger().info("Create custom mapping alias='{}' url='{}' expiredAt='{}' active='{}'",alias,url,expiredAtOrNull,activeOrNull);varresult=UrlValidator.validate(url);if(!result.valid()){thrownewIllegalArgumentException("Invalid URL: "+result.message());}if(alias!=null&&!alias.isBlank()){varvalidate=AliasPolicy.validate(alias);if(!validate.valid()){varreason=validate.reason();thrownewIllegalArgumentException(reason.defaultMessage);}}varshortenRequest=newShortenRequest(url,alias,expiredAtOrNull,activeOrNull);Stringbody=toJson(shortenRequest);logger().info("createCustomMapping - body - '{}'",body);URLshortenUrl=serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL();logger().info("connecting to .. shortenUrl {} (custom)",shortenUrl);HttpURLConnectionconnection=(HttpURLConnection)shortenUrl.openConnection();connection.setRequestMethod("POST");connection.setDoOutput(true);connection.setRequestProperty(CONTENT_TYPE,JSON_CONTENT_TYPE);try(OutputStreamos=connection.getOutputStream()){os.write(body.getBytes(UTF_8));}intstatus=connection.getResponseCode();logger().info("Response Code from Server - {}",status);if(status==200||status==201){try(InputStreamis=connection.getInputStream()){StringjsonResponse=newString(is.readAllBytes(),UTF_8);logger().info("createCustomMapping - jsonResponse - {}",jsonResponse);ShortUrlMappingshortUrlMapping=fromJson(jsonResponse,ShortUrlMapping.class);logger().info("shortUrlMapping .. {}",shortUrlMapping);returnshortUrlMapping;}}if(status==409){finalStringerr=readAllAsString(connection.getErrorStream());thrownewIllegalArgumentException("Alias already in use: '"+alias+"'. "+err);}if(status==400){finalStringerr=readAllAsString(connection.getErrorStream());thrownewIllegalArgumentException("Bad request: "+err);}thrownewIOException("Server returned status "+status);}

The simple variant ofcreateCustomMapping serves as a convenience method: it accepts only an alias and a URL and delegates to the extended version, setting expiration date and activity status to zero. This keeps the API lean for simple use cases, while allowing complete control over the shortlink via the overloaded method.

The extended method first assumes responsibility for validating the input data.UrlValidator.validate(url) checks whether the specified destination URL meets the expected criteria. If this is not the case, anIllegalArgumentException is thrown with a comprehensible error message. If an alias is set, the alias policy is then checked. This ensures that custom shortcodes comply with the set rules and, for example, do not contain unwanted special characters or prohibited patterns.

If the URL and alias are valid, aShortenRequest is created that includesexpiredAtOrNullandactiveOrNull, in addition to URL and alias. In this way, the user can control whether the shortlink has an expiration date and whether it should be initially active or inactive when creating it. The request is then serialised in JSON format and sent to thePATH_ADMIN_SHORTEN endpoint.

The HTTP configuration follows the familiar pattern: APOSTrequest is created with a JSON body, theContent-Type is set accordingly, and the body is transmitted via the OutputStream. The server’s response is first checked against the HTTP status code. If successful (200 or201), the client reads the response body, converts it to aShortUrlMapping object, and returns it to the user.

Two special paths are provided for error cases: If the alias reservation fails because the alias is already assigned (409 Conflict), anIllegalArgumentException is thrown with a clear description. General validation errors (400 Bad Request) also throw a meaningfulIllegalArgumentException. All other unexpected status codes result in anIOException that includes the exact status code and error message from the server.

Overall, the advancedcreateCustomMapping method fits seamlessly into the existing API design. It allows users to create shortlinks in one step with an alias, expiration date, and activity status, combining strict validation with precise, predictable error handling.

Adjustments toedit(...) – Processing with activity status

In addition to creating new shortlinks, users often need to customise existing listings. This may involve updating the target URL, adjusting an expiration date, or, in the context of the new functionality, activating or deactivating an existing shortlink.

To cover these requirements, the existingedit(...)method of the Java client has been extended. It now also supports the optional activity status parameter, so the user no longer needs to change this value via a downstream toggle API; they can control it directly during editing.

This approach facilitates workflows in which multiple properties of a shortlink are edited simultaneously. Instead of making various API calls in sequence, the entire change process can be combined into a single edit operation. This means that the API remains efficient, consistent and can be easily integrated into various application scenarios – from manual editing to the UI to automated processes in scripts or backend systems.

In the next step, we will discuss the implementation of the extendededit(...)method.

The following implementation comes from theURLShortenerClient and shows how the activity state was incorporated into the editing process:

javaCopy

publicbooleanedit(StringshortCode,StringnewUrl,InstantexpiresAtOrNull,BooleanactiveOrNull)throwsIOException{logger().info("Edit mapping alias='{}' url='{}' expiredAt='{}' active='{}'",shortCode,newUrl,expiresAtOrNull,activeOrNull);if(shortCode==null||shortCode.isBlank()){thrownewIllegalArgumentException("shortCode must not be null/blank");}if(newUrl==null||newUrl.isBlank()){thrownewIllegalArgumentException("newUrl must not be null/blank");}finalURIuri=serverBaseAdmin.resolve(PATH_ADMIN_EDIT+"/"+shortCode);finalURLurl=uri.toURL();logger().info("edit - {}",url);finalHttpURLConnectioncon=(HttpURLConnection)url.openConnection();con.setRequestMethod("PUT");con.setDoOutput(true);con.setRequestProperty(CONTENT_TYPE,JSON_CONTENT_TYPE);con.setRequestProperty(ACCEPT,APPLICATION_JSON);con.setConnectTimeout(CONNECT_TIMEOUT);con.setReadTimeout(READ_TIMEOUT);finalShortenRequestreq=newShortenRequest(newUrl,shortCode,expiresAtOrNull,activeOrNull);finalStringbody=toJson(req);logger().info("edit - request body - '{}'",body);try(OutputStreamos=con.getOutputStream()){os.write(body.getBytes(UTF_8));}finalintcode=con.getResponseCode();logger().info("edit - responseCode {}",code);if(code==200||code==204||code==201){drainQuietly(con.getInputStream());returntrue;}if(code==404){logger().info("shortCode not found.. {}",shortCode);drainQuietly(con.getErrorStream());returnfalse;}finalStringerr=readAllAsString(con.getErrorStream());thrownewIOException("Unexpected response: "+code+", body="+err);}

The method accepts four parameters: the shortcode to change, the new destination URL, an optional expiration date, and an optional activity status. Right from the start, we check whether the necessary fields – especiallyshortCode andnewUrl – are valid. An IllegalArgumentException immediately catches invalid inputs.

The request is then sent as aPUT to the corresponding edit endpoint. The payload consists of aShortenRequestthat contains all editable attributes of a shortlink – including the activity specificationactiveOrNull. This allows the user to update the URL, expiration date, and activity status in a single process.

The client then processes the HTTP response. Success cases (200,201,204) returntrue, while a404 clearly indicates that the shortlink does not exist. Unexpected status codes cause anIOException, which allows the user to provide precise fault diagnoses.

This extension makes theedit(...)method fully capable of updating all relevant properties of a shortlink in a single step, including the activity state, which was previously only controllable via a separate toggle endpoint.

User interactions and UI logic in the Vaadin interface

After the previous chapters covered server-side REST handlers, client APIs, and persistence, this chapter focuses on the application’s UI layer. TheOverviewView is the central administration tool for users to search, filter, edit, and manage shortlinks in bulk.

Chapter 6, therefore, sheds light on the most critical user interactions in the frontend, in particular:

• The interaction ofVaadin-Grid ,CallbackDataProvider and dynamic filter parameters
• the integration ofsingle actions (e.g. activating/deactivating a link by clicking on an icon)
• Implement more complexbulk operations , including dialogues, error handling, and visual feedback
• The connection between UI inputs and the REST endpoints via theURLShortenerClient

Switching the active state directly in the grid

To allow users to change the active or inactive status of a shortlink not only via the API, but also conveniently in the user interface, the grid of the administration view has been extended. The goal is to create the most direct, intuitive, and low-risk way possible to switch the status of a shortlink without opening separate dialogues or editing masks.

At the centre of each table row is a clearly recognisable visual button. With a single click, the user can toggle a shortlink’s status between active and inactive. The interface immediately signals the current state of the shortlink and the action that will be taken when clicked.

This enlargement has three main objectives:

• Speed – frequent switching does not require navigation into additional masks.
• Transparency – the user can see at all times which shortlinks are currently active.
• Robustness – possible error situations are clearly communicated and do not affect the rest of the application.

The core of the implementation lies in configuring the grid and the new “Active” column. The relevant snippet from theOverviewView looks like this:

javaCopy

grid.addComponentColumn(m->{Iconicon=m.active()?VaadinIcon.CHECK_CIRCLE.create():VaadinIcon.CLOSE_CIRCLE.create();icon.setColor(m.active()?"var(--lumo-success-color)":"var(--lumo-error-color)");icon.getStyle().set("cursor","pointer");icon.getElement().setProperty("title",m.active()?"Deactivate":"Activate");icon.addClickListener(_->{booleannewValue=!m.active();try{urlShortenerClient.toggleActive(m.shortCode(),newValue);Notification.show("Status updated",2000,Notification.Position.TOP_CENTER);safeRefresh();}catch(Exceptionex){Notification.show("Error updating active status: "+ex.getMessage(),3000,Notification.Position.TOP_CENTER);}});returnicon;}).setHeader("Active").setKey("active").setAutoWidth(true).setResizable(true).setSortable(true).setFlexGrow(0);

Instead of a plain text field, acomponent column is used here, displaying a separate icon for each row. The choice of icon depends directly on the current activity status of the respective shortlink:

• Ifm.active() istrue , aCHECK_CIRCLEicon is displayed.
• Ifm.active()isfalse , aCLOSE_CIRCLE icon is used.

The colour scheme (success for active,error for inactive) allows the user to recognise the state at a glance. In addition, thetitle attribute on the icon tells you which action is triggered by a click (“Activate” or “Deactivate”).

The toggle mechanism is implemented in the icon’s click listener. When clicked, the desired new status is first calculated:

javaCopy

booleannewValue=!m.active();

The Java client is then called, which delegates the state change to the server via the REST API:

javaCopy

urlShortenerClient.toggleActive(m.shortCode(),newValue);

If the call succeeds, the user receives a short, unobtrusive confirmation notification, and the grid is reloaded viasafeRefresh(). This means that subsequent changes (e.g. filtering by active status) are also displayed correctly immediately.

Error handling follows the same pattern as in the client API: If an exception occurs – whether due to network problems, unexpected HTTP status codes or server errors – it is communicated in the UI via a clearly visible notification:

javaCopy

}catch(Exceptionex){Notification.show("Error updating active status: "+ex.getMessage(),3000,Notification.Position.TOP_CENTER);}

For the user, this means that the active status of a shortlink can be changed directly in the overview with a click. The UI combines clear visual cues (icon, colour, tooltip) with immediate feedback and robust error handling. This reduces the need for separate processing dialogues and makes typical administrative tasks around active/inactive much more efficient.

Filter by active and inactive status

In addition to the ability to switch a shortlink’s active status directly in the grid, the user interface has been enhanced with a precise filter system. This allows the user to search specifically for active, inactive, or all shortlinks – without manual searches or complex queries.

The goal is to provide the user with a tool that allows them to control the visibility of entries flexibly. This improves both the clarity of large datasets and the efficiency of routine administrative tasks, such as detecting expired or disabled shortlinks.

A unified selection element controls the new filter and affects both data retrieval and paging. If the user sets the filter to Active, Inactive, or Not set, the query parameters are updated to load and display only relevant records in the grid.

Central to this is theSelectfield for the active status, which is defined in theOverviewView:

xmlCopy

private final Select<ActiveState> activeState = new Select<>();

This UI element is configured in the search bar and displayed alongside other search and paging elements. Initialisation is done in thebuildSearchBar() block:

javaCopy

activeState.setLabel("Active state");activeState.setItems(ActiveState.values());activeState.setItemLabelGenerator(state->switch(state){caseACTIVE->"Active";caseINACTIVE->"Inactive";caseNOT_SET->"Not set";});activeState.setEmptySelectionAllowed(false);activeState.setValue(ActiveState.NOT_SET);HorizontalLayouttopBar=newHorizontalLayout(globalSearch,searchScope,pageSize,activeState,resetBtn);

This gives the user a clearly labeled drop-down selection with three states:

• Active – only active shortlinks
• Inactive – only inactive shortlinks
• Not set – no filtering by active status

The filter is set to NOT_SET by default , so all shortlinks are displayed first. The ItemLabelGenerator function determines the label displayed in the drop-down for each enum value.

For the filter to be functionally effective, the view reacts to changes in the selection field. Acorresponding listener is registered in the addListeners() block:

javaCopy

activeState.addValueChangeListener(_->{currentPage=1;safeRefresh();});

As soon as the user changes the active state, the current page is reset to 1 and the data provider is reloaded. This makes the filter changes directly visible and prevents the user from being on an invalid page (for example, if there are fewer entries due to filtering).

The actual connection to the REST API is created in the buildFilter(...)method block that creates aUrlMappingListRequest from the UI :

javaCopy

privateUrlMappingListRequestbuildFilter(Integerpage,Integersize){UrlMappingListRequest.Builderb=UrlMappingListRequest.builder();ActiveStateactiveStateValue=activeState.getValue();logger().info("buildFilter - activeState == {}",activeStateValue);if(activeStateValue!=null&&activeStateValue.isSet()){b.active(activeStateValue.toBoolean());}// ... Other filters (codePart, urlPart, time periods, sorting, paging)if(page!=null&&size!=null){b.page(page).size(size);}varfilter=b.build();logger().info("buildFilter - {}",filter);returnfilter;}

Here, the enum value of theactiveStateselect is read and, if it is considered “set,” converted to a Boolean (true for active, false for inactive). The request builder takes over this value and later transmits it to the server by the client (URLShortenerClient) as a query parameter. If the state isNOT_SET, noactive value is set, so there is no active-status restriction on the server side.

Together, these building blocks form a consistent filter concept:

• TheSelect<ActiveState>provides a clear, three-level selection.
• The ValueChangeListener ensures that filter changes take effect immediately.
• buildFilter(...) translates the UI selection into a typed request to the backend API.

For users, this creates a seamless interaction: Switching from “Active” to “Inactive” immediately leads to only the corresponding shortlinks appearing in the grid – without this logic having to be duplicated in the frontend or filtered on the client side.

Mass operations based on active status.

In addition to single editing, the interface offers convenient bulk operations that let users enable or disable multiple shortlinks at once. This is especially helpful in scenarios where large amounts of data need to be managed, such as temporarily shutting down campaign links or reactivating an entire group of previously disabled shortlinks.

In this extension, bulk handling has been designed to integrate seamlessly with the existing grid and the integrated selection mechanisms. The user can select as many rows as they want in the grid and then trigger appropriate actions via clearly recognisable buttons in the bulk bar at the bottom of the screen.

The process always follows the same pattern:

1. The user marks the desired short links.
2. A confirmation dialogue ensures that the mass change is deliberately triggered.
3. The action is performed for each selected entry, regardless of potential errors.
4. The result is displayed to the user in a compact success/failure overview.

This approach enables robust yet user-friendly bulk editing. The interface remains responsive, and error handling ensures that a failure on a single shortlink does not affect the entire operation.

The following snippet is taken directly from theOverviewView and shows the central method that enables or disables a set of shortlinks in one pass:

xmlCopy

private void bulkSetActive(Set<ShortUrlMapping> selected, boolean activate) { int success = 0; int failed = 0; for (var m : selected) { try { var ok = urlShortenerClient.toggleActive(m.shortCode(), activate); if (ok) { success++; } else { failed++; } } catch (IOException ex) { logger().error("Toggle active state failed for {}", m.shortCode(), ex); failed++; } } grid.deselectAll(); safeRefresh(); var actionLabel = activate ? "Activate" : "Deactivate"; Notification.show( actionLabel + " – Success: " + success + " • Failed: " + failed );}

This method is central to mass activation. It receives the shortlinks currently selected in the grid, as well as the target status (activate = true orfalse). For each entry, the corresponding request is sent to the server API via the Java client. The approach is deliberately designed to be fault-tolerant: a failure of one entry does not affect the rest of the processing.

The UI is not updated until the loop is complete. This keeps the operation clear and efficient, even with many shortlinks. The result is displayed to the user in a compact summary.

The user triggers this operation via a confirmation dialogue. This is implemented inconfirmBulkSetActiveSelected(boolean activate ):

javaCopy

privatevoidconfirmBulkSetActiveSelected(booleanactivate){varselected=grid.getSelectedItems();if(selected.isEmpty()){Notification.show("No entries selected");return;}varverb=activate?"activate":"deactivate";varverbCap=activate?"Activate":"Deactivate";Dialogdialog=newDialog();dialog.setHeaderTitle(verbCap+" all "+selected.size()+" short links?");dialog.add(newText("This will "+verb+" all selected short links. "+"They will be "+(activate?" active":"inactive")+" afterwards."));Buttoncancel=newButton("Cancel",_->dialog.close());Buttonconfirm=newButton(verbCap+" All",_->{dialog.close();bulkSetActive(Set.copyOf(selected),activate);});confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY);dialog.getFooter().add(newHorizontalLayout(cancel,confirm));dialog.open();}

This dialogue ensures that mass changes are not accidentally triggered. This is especially relevant for deactivating actions, as a deactivated shortlink no longer redirects.

The dialogue is activated via the two buttons in the bulk bar:

javaCopy

bulkActivateBtn.addClickListener(_->confirmBulkActivateSelected());bulkDiActivateBtn.addClickListener(_->confirmBulkDeactivateSelected());

The associated wrapper methods are:

javaCopy

privatevoidconfirmBulkActivateSelected(){confirmBulkSetActiveSelected(true);}privatevoidconfirmBulkDeactivateSelected(){confirmBulkSetActiveSelected(false);}

This means that mass editing is fully integrated into the user interface: selection, confirmation, execution and result feedback follow a clear, consistent process model. For users, an intuitive workflow makes administrative tasks much easier.

Redirect behaviour for end users.

With the introduction of the new active/inactive mechanism and improved handling of expiration times (expiresAt), the URL shortener’s behaviour when calling a shortlink changes fundamentally. While users previously received only a redirect or a generic error, the system now provides more granular HTTP status codes that enable precise conclusions about a link’s status.

Expired Shortlinks (expiresAt) → 410 Gone

If a shortlink has an expiration date and has passed, the user must not be redirected to the original URL when accessed. Instead, the system must clearly signal that this shortlink exists, but is no longer valid. This is precisely what the HTTP status code410 Gone is used for.

The status code 410 indicates “permanently removed” and makes it unmistakably clear that the shortlink was previously active but is now deliberately unavailable. Unlike 404 Not Found, which means that a resource does not exist, 410 conveys a clear semantic meaning:this shortlink has expired and will never be valid again.

For the user, this means the call results in clear error behaviour that remains comprehensible. For developers, on the other hand, this mechanism creates transparency, as they can clearly distinguish between three situations:

• A shortlink does not exist → 404 Not Found
• A shortlink exists, but is disabled → 404 Not Found
• A shortlink exists and has expired →410 Gone

The consistent use of the 410 status code also enables better monitoring and cleaner automation processes. Systems such as API gateways, SEO tools, crawlers, or CI/CD pipelines can capture the exact behaviour and thus more precisely explain why a redirect does not occur.

Disabled (active = false) → 404 Not Found

Another central component of the new active/inactive model is how deactivated shortlinks are handled. While expired links are signalled with the HTTP status code410 Gone , the system deliberately uses a different status code for disabled links:404 Not Found.

This difference is not accidental, but follows clear safety and operational considerations. A deactivated shortlink should behave as if it no longer exists for the end user, although it remains fully present, visible, and manageable internally.

This creates a behaviour ideal for maintenance phases, temporary blockades, campaign stops, or safety-related shutdowns. Developers and administrators retain complete control over the dataset without inadvertently revealing internal state information.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-de-activate-mappings-part-1/Fri, 19 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-de-activate-mappings-part-1/Why an active/inactive model for shortlinks? For many users – especially those who work in the field of software development – shortlinks are much more than simple URL shorteners. They act as flexible routing mechanisms for campaigns, feature controls, test scenarios, and internal tools. The requirements for transparency, controllability and clean lifecycle management are correspondingly high.<![CDATA[

Why an active/inactive model for shortlinks?

For many users – especially those who work in the field of software development – shortlinks are much more than simple URL shorteners. They act as flexible routing mechanisms for campaigns, feature controls, test scenarios, and internal tools. The requirements for transparency, controllability and clean lifecycle management are correspondingly high.

The URL shortener receives an active/inactive model that directly supports these claims. A user can now specify, for each shortlink, whether it should be active, temporarily deactivated, or no longer allowed to reach users due to a planned expiration. At the same time, a clear distinction is made between a disabled and an expired shortlink – both in the UI and in the HTTP behaviour.

For the user, this creates a noticeable gain in control. An entire campaign can be taken offline with a single click, without losing data or having to edit multiple entries manually. The REST API or the Java client can be used to automatically control activity status externally, enabling integrations with existing development and deployment processes.

This introduction outlines the motivation for the new model. The following chapters detail how the UI, API, data model, and redirect logic have been extended, and how users benefit from the new active/inactive mechanism.

The source code for this project‘s status can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-10

Architecture overview of changes

The new active/inactive model does not work in isolation within a single module; it spans multiple layers of the application. This creates consistent behaviour for the user – regardless of whether a shortlink is edited via the UI, automatically managed via the API or accessed directly in the browser. To understand this interaction, it is worth examining the key areas where adjustments have been made.

The basis is the extended data model, which now includes an additional attribute to store the activity status. This feature serves as the common denominator for all shortlink-related operations. As a result, both the REST endpoints and the internal Java client have been extended to reliably transmit activity status when creating, modifying, or querying a shortlink.

These structural changes are also reflected in the user interface. The activity status can now be set directly when creating a shortlink, in the overview table, or in the detail dialogue. Actions such as activating or deactivating multiple entries simultaneously use the exact mechanisms as the API, creating a consistent application experience.

Last but not least, the redirect behaviour has also been adjusted. While expired shortlinks are clearly indicated, deactivated shortlinks are not. In this way, the user can not only better understand why a shortlink is unreachable, but also receive more precise feedback from both client and monitoring perspectives.

This architectural overview shows that the active/inactive model is not a one-off feature, but a consistent design principle across all layers of the system. In the following chapters, the respective areas are considered in detail and technically explained.

The Advanced Data Model

Before the active/inactive model becomes visible in the user interface, REST API, or redirect behaviour, the basic structure must be defined in the data model. Chapter 3 focuses on this basis and shows how a shortlink’s activity status is anchored in the core of the system.

The data model is the central place where all relevant information converges into a shortlink. Every user action – be it creating, editing, filtering or forwarding a shortlink – sooner or later falls back on this structure. Therefore, it is crucial to introduce an activity status that is consistently and reliably available.

This chapter explains how the newactive attribute is mapped to domain objects, how it is transported in DTOs, and the serialisation and backwards-compatibility considerations involved. This clean anchoring in the data model makes the active/inactive model a stable part of the entire application, on which all further technical enhancements are based.

Newactiveflag in the core domain model.

The introduction of the active/inactive model begins at the core domain model level. This defines how a shortlink stores its state and what information is transmitted within the application. The new active attribute lets users check at any time whether a shortlink is currently in use or has been deliberately deactivated.

The central element of this change is the extension of theShortUrlMapping class. It outlines the essential characteristics of a short link, including the alias, the original destination URL, and the expiration dates. The new attribute enables the unique determination of the activity status within this structure.

In the original implementation, the relevant snippet looks like this:

javaCopy

privatebooleanactive;publicbooleanactive(){returnactive;}publicShortUrlMappingwithActive(booleanactive){returnnewShortUrlMapping(this.shortCode,this.originalUrl,this.createdAt,this.expiresAt,active);}

This enhancement ensures that activity status is reliably available both in memory and in all downstream layers of the system. The model remains unchanged. The status is not changed in an existing object; instead, itis created as a new instance using the withActive method. This prevents unintended side effects and supports consistent, deterministic data behaviour.

This clear separation between active and inactive states provides a stable foundation for UI, server, and redirect logic. In the following subchapters, we will discuss how this information is mapped to the DTOs and the consequences for serialisation and transport behaviour.

Impact on DTOs (ShortenRequest,ShortUrlMapping)

The active/inactive model not only affects the domain core but must also be transported cleanly across the application boundary. Data Transfer Objects (DTOs) are crucial for this. They serve as the interface between the user interface, REST API, and Java client, ensuring that the activity status of a shortlink is transmitted correctly.

With the expansion of the data model, the DTOShortUrlMapping also receives an additional attribute that reflects the status of a shortlink. This value is used both in communication from the server to the UI and between the server and the client. A relevant excerpt from the original implementation looks like this:

javaCopy

publicrecordShortUrlMapping(StringshortCode,StringoriginalUrl,InstantcreatedAt,InstantexpiresAt,booleanactive){}

This extended DTO enables the user to view activity status in the UI and allows external systems to read the status and respond accordingly. By using a record, structure and serialisation remain lean and clearly defined.

The request to create or change a shortlink has also been adjusted. TheShortenRequest now also includes information on whether a shortlink should be made directly active or initially deactivated. This allows the user to control how the shortlink behaves during creation.

An excerpt from the corresponding record:

javaCopy

publicrecordShortenRequest(StringshortCode,StringoriginalUrl,InstantexpiresAt,Booleanactive){}

At this point, it is evident that the request uses aBoolean, whereas the mapping itself uses a primitiveBoolean. This difference is deliberate: the value in the request may also benull if the user does not want to set an explicit state. In this case, the server logic decides on a default value. Mapping, on the other hand, always has a defined state, eliminating ambiguity.

Extending DTOs ensures the active/inactive model is uniformly available across all system boundaries. It forms the basis for the UI, API and client to be informed identically about the status of a shortlink, thus creating a consistent user experience.

Serialisation and backward compatibility

For the active/inactive model to function reliably, the newactive value must not only be present in the internal data model but also correctly serialised and transported across different components. The extension of serialisation affects two areas in particular: JSON communication via REST and data exchange between the various application modules.

By using Java records in the DTOs, serialisation is primarily handled by the chosen JSON library. Once theactive attribute is defined in the record constructors, it is automatically included in the JSON representation. This makes integration straightforward and minimises the need for additional configuration.

An example of the resulting JSON structure of a shortlink as served via the REST API:

jsonCopy

{"shortCode":"abc123","originalUrl":"https://example.com","createdAt":"2025-05-20T10:15:30Z","expiresAt":"2025-06-01T00:00:00Z","active":true}

The JSON structure makes it clear that the activity state is immediately visible to the user and can be retrieved without any additional logic. This increase in transparency is central to the new model.

An essential aspect of this change is backward compatibility. Systems or components that use older versions of the API and are not aware of theactive field can continue to interact without issues. JSON consumers typically ignore unknown fields to avoid affecting older clients. At the same time, the server can set a sensible default value for requests that do not specify an active value – typicallytrue.

This combination of clear serialisation and high backward compatibility ensures that the active/inactive model can be integrated into existing systems without disruption. At the same time, further development remains open to future extensions, such as more differentiated activity states or audit information.

Enhancements to the Admin REST API

The active/inactive model only unfolds its full effect through the extensions of the administrative REST API. This establishes the connection between the data model, the internal business logic, and the various clients that create, edit, or automatically manage shortlinks. To enable users to effectively use the new activity state, several API endpoints had to be adapted or supplemented.

Chapter 4 examines these extensions in detail. It shows how the new toggle endpoint enables targeted switching of activity status, how inactive entries can be queried via a dedicated list endpoint, and how the ActiveState filter extends existing query operations. It also explains how differentiated HTTP status codes clearly convey the semantic meaning of a deactivated or expired shortlink.

These API customisations create a coherent, well-integrated interface that can map all aspects of the active/inactive model, whether the requests come from a user interface, a Java client, or an automated system.

New Toggle Endpoint

To make the active/inactive shortlink model usable across the application, the server component needs a well-defined mechanism to change the activity status of an existing shortlink. For this purpose, a new toggle endpoint was introduced and implemented on the server side byToggleActiveHandler. It forms the basis for allowing users to adjust the activity state of a shortlink directly from the REST API – whether from the user interface, automations, or external systems.

The new endpoint has a clear purpose: it enables the targeted switching of the activity status of a specific shortlink. The user can specify, via a simple request, whether a shortlink should be activated or deactivated without changing any other mapping properties. This not only makes the operation more efficient, but also less error-prone, as there is no need for full update payloads.

This endpoint integrates seamlessly with the existing API structure and uses the same transport models and validation mechanisms as other administrative operations. At the same time, it provides a clear separation between changes of activity status and other editing operations, simplifying both implementation and use.

The core component of this endpoint is theToggleActiveHandler. It encapsulates the HTTP-specific processing and delegates the actual state change to theUrlMappingStore behind it:

xmlCopy

public class ToggleActiveHandler implements HttpHandler, HasLogger { private final UrlMappingStore store; public ToggleActiveHandler(UrlMappingStore store) { this.store = store; } @Override public void handle(HttpExchange ex) throws IOException { logger().info("handle ... {} ", ex.getRequestMethod()); if (! RequestMethodUtils.requirePut(ex)) return; try { final String body = readBody(ex.getRequestBody()); ToggleActiveRequest req = fromJson(body, ToggleActiveRequest.class); var shortCode = req.shortCode(); if (isNullOrBlank(shortCode)) { writeJson(ex, BAD_REQUEST, "Missing 'shortCode'"); return; } boolean newActiveValue = req.active(); logger().info("Toggling active status for {} to {}", shortCode, newActiveValue); Result<ToggleActiveResponse> mapping = store.toggleActive(shortCode, newActiveValue); if (mapping.isPresent()) { logger().info("Toggling active status successfully"); writeJson(ex, OK, toJson(mapping.get())); } else { Mapping. ifFailed(failed -> logger().info("Toggling active status failed: {}", failed)); writeJson(ex, BAD_REQUEST, "Toggling active status failed"); } } catch (RuntimeException e) { logger().warn("catch - {}", e.toString()); writeJson(ex, INTERNAL_SERVER_ERROR); } finally { logger().info("ToggleActiveHandler .. finally"); ex.close(); } }}

The handler implements the HttpHandler interface and integrates project-specific logging viaHasLogger. In the constructor, the dependency is injected into theUrlMappingStore, which is later used to execute the logic that changes the state.

The main logic is in thetry block. First, the request body iscompletely read in viareadBody(ex.getRequestBody()) and convertedto an instance ofToggleActiveRequest usingfromJson. This Request object contains the two relevant pieces of information: theshortCode and the newactive value. It then checks whether theshortcode is empty or null. In this case, the handler responds immediately with a400 Bad Request and a clear error message, preventing the store layer from encountering invalid data.

If theshortcode is valid, the desired new activity value is extracted and logged. The actual status change occurs via “store.toggleActive(shortCode, newActiveValue)". The result is encapsulated in aResult<ToggleActiveResponse> to ensure both success and failure cases are handled consistently.

In case of success (mapping.isPresent()), another success entry is written to the log, and the updated display of the shortlink is returned to the user as JSON with the HTTP status200 OK. If the store does not return a result, the cause of the error is logged viaifFailed, and the user is sent a response with a400 Bad Request status code and a generic error message. This clearly defines the error behaviour without revealing too many details internally.

If unexpected runtime errors occur in the handler itself, thecatch block catches the exception, logs a warning, and returns a500 Internal Server Error. Thefinally block ensures that the connection is closed cleanly viaex.close() in all cases and that a corresponding log entry is created. Overall, this creates a robust, clearly structured endpoint that reliably and comprehensibly updates a shortlink‘s activity status.

Query inactive links

In addition to targeting the activity status of an individual shortlink, the user needs the ability to view collected inactive entries. This provides an overview of shortlinks that have been deliberately or automatically disabled and are currently no longer redirecting.

For this purpose, a special API endpoint was introduced that returns only inactive shortlinks. This endpoint complements the general list endpoint and provides a clear, filtered view of all shortlinks with the “inactive” status.

The focus of this endpoint is on visibility and control. Users can quickly see which shortlinks are currently disabled without having to implement their own filtering logic or evaluate the complete list of shortlinks. This is a notable advantage, especially in larger installations or automated workflows, as inactive entries often require separate management processes.

The endpoint integrates seamlessly with the existing administrative API structure and uses the same data models and return formats as other query operations. It ensures that information about inactive shortlinks is consistently retrievable across all UIs and clients.

The technical implementation of querying inactive shortlinks is integrated into the existing list handler. The following excerpt shows theListHandler responsible for this, which provides different list variants, including the inactive entries, via a common endpoint:

xmlCopy

public class ListHandler implements HttpHandler, HasLogger { private final UrlMappingLookup store; public ListHandler(UrlMappingLookup store) { this.store = store; } SNIPP @Override public void handle(HttpExchange ex) throws IOException { if (! RequestMethodUtils.requireGet(ex)) return; final String path = ex.getRequestURI().getPath(); logger().info("List request: {}", path); String responseJson; if (path.endsWith(PATH_ADMIN_LIST_ALL)) { responseJson = listAll(); } else if (path.endsWith(PATH_ADMIN_LIST_EXPIRED)) { responseJson = listExpired(); } else if (path.endsWith(PATH_ADMIN_LIST_ACTIVE)) { responseJson = listActive(); } else if (path.endsWith(PATH_ADMIN_LIST_INACTIVE)) { responseJson = listInActive(); } else if (path.endsWith(PATH_ADMIN_LIST)) { responseJson = listFiltered(ex); } else { logger().info("undefined path {}", path); ex.sendResponseHeaders(404, -1); return; } writeJson(ex, OK, responseJson); } private String listInActive() { final Instant now = Instant.now(); return filterAndBuild("inactive", m -> { if (!m.active()) return true; return m.expiresAt() .map(exp -> exp.isBefore(now)) .orElse(false); }); } private String filterAndBuild(String mode, Predicate<ShortUrlMapping> predicate) { final Instant now = Instant.now(); final var data = store .findAll() .stream() .filter(predicate) .map(m -> toDto(m, now)) .toList(); return JsonUtils.toJsonListing(mode, data.size(), data); } private Map<String,String> toDto(ShortUrlMapping m, Instant now) { boolean expired = m. expiresAt() .map(t -> t.isBefore(now)) .orElse(false); return Map.of( "shortCode", m.shortCode(), "originalUrl", m.originalUrl(), "createdAt", m.createdAt().toString(), "expiresAt", m.expiresAt().map(Instant::toString).orElse(""), "active", m.active() + "", "status", expired ? "expired" : "active" ); }}

TheListHandler, like the toggle handler, implements theHttpHandler interface and usesHasLogger for consistent logging. AUrlMappingLookup isinjected into the constructor and used for the data queries. Thehandlemethod handles routing based on the request path: Depending on whether the request ends inPATH_ADMIN_LIST_ALL,PATH_ADMIN_LIST_ACTIVE, orPATH_ADMIN_LIST_INACTIVE, a specialised list method is called.

The listInActive()method is responsible for querying inactive shortlinks. It determines the current time and callsfilterAndBuild("inactive", ...) with a predicate that selects the desired entries. A shortlink is considered inactive if it is either explicitly marked as inactive (!m.active()) or if it is active but has already expired. The combination of the active flag and the expiration date allows a clean separation between currently usable and no longer usable entries.

ThefilterAndBuild method reads all entries from theUrlMappingLookup, applies the passed predicate, and converts the remaining mappings into a DTO-like structure. For this purpose,toDto is used to create a map of the most essential attributes from eachShortUrlMapping. In addition toshortCode,originalUrl,createdAt, andexpiresAt, theactivestatus and a derived field,status, are set here, which distinguishes betweenexpired andactive. This additional information makes it easier for the user to read the entry status directly from the response.

This structure allows queries for inactive links to fit seamlessly into the existing list concept. Instead of introducing a completely independent endpoint, the existing infrastructure is used and expanded to include a targeted view of inactive entries. The result is a precise, reusable mechanism that always gives users a complete overview of all currently inactive shortlinks.

Adjustments in the list request (ActiveState filter)

To allow users to search specifically for active, inactive, or all shortlinks, a dedicated activity status filter has been added to the existing list mechanism. This so-calledActiveState filter complements the previous filter criteria, such as text search, sorting, time periods, or pagination, and enables precise control of the search results via the API.

This extension provides the user with a flexible yet clearly defined way to set the desired status directly via a query parameter. Instead of retrieving the entire list of shortlinks and then filtering it on the client side, the request can now be restricted directly on the server. This saves resources, reduces unnecessary data transfers, and ensures consistent search results.

The ActiveState filter considers three possible states:

• Active – The shortlink is active and, if there is an expiration date, has not yet expired.
• Inactive - The shortlink has been disabled by the user or has expired.
• Not set – The user does not provide a status indication, so the filter is not applied, and all relevant entries are considered.

This distinction is communicated via the active query parameter, which is passed to the standard list endpoint in a GET request. The server evaluates this parameter and creates a correspondingUrlMappingFilter object based on it, which controls entry selection.

The technical implementation of this filter logic is handled in the existingListHandler, which already handles filtering and sorting shortlinks. The following excerpt shows how to process the active parameter and embed it in theUrlMappingFilter:

kotlinCopy

varfilter=UrlMappingFilter.builder().codePart(first(query,"code")).urlPart(first(query,"url")).createdFrom(parseInstant(first(query,"from"),true).orElse(null)).createdTo(parseInstant(first(query,"to"),false).orElse(null)).active(parseBoolean(first(query,"active")).orElse(null)).offset(offset).limit(size).sortBy(sortBy.orElse(null)).direction(dir.orElse(null)).build();

The central point is the line:

javaCopy

.active(parseBoolean(first(query,"active")).orElse(null))

Here, the query parameteractive is read, interpreted as an optional boolean. The process in detail:

1. **first(query, "active")** reads the first value of the query parameteractive – something like:
   1. active=true
   2. active=false
   3. or the parameter is missing completely.
2. **parseBoolean(...)** converts this value to anOptional<Boolean> . This means that invalid or missing values can also be caught cleanly.
3. **.orElse(null)** ensures that if input is missing or cannot be interpreted, the valuenull is transferred to the filter.

This results in the following meaning:

• active=true → only active shortlinks
• active=false → only inactive shortlinks
• no parameter → no restriction

The UrlMappingLookup then processes the resulting UrlMappingFilter:

kotlinCopy

inttotal=store.count(filter);varresults=store.find(filter);

These methods replace the application of the filter to the saved shortlinks. By passing through theactivestate, the lookup layer can precisely select the entries corresponding to the desired activity state.

Finally, the results are transferred to the DTO structure and returned as a paginated JSON response:

javaCopy

varitems=results.stream().map(m->toDto(m,now)).toList();returntoJsonListingPaged("filtered",items.size(),items,page,size,total,sortBy.orElse(null),dir.orElse(null));

With this extension, the ActiveState filter fits seamlessly into the existing filter system. Users can now target active or inactive shortlinks without additional endpoints or separate calls. This shows how ActiveState is processed and how it affects the final JSON result.

HTTP status code semantics: 410 vs. 404

With the introduction of the active/inactive model, the behaviour of the forwarding logic also changes. For the user, the state of a shortlink must be clearly and unambiguously communicated to the outside world – especially if a redirect cannot be done as expected. For this reason, the system uses two distinct HTTP status codes to distinguish expired and disabled shortlinks.

A shortlink can no longer be reached for two reasons:

1. It has expired. The expiration date has passed, and the shortlink is no longer valid as planned.
2. It has been disabled. The user has manually disabled the shortlink, regardless of the expiration date.

Although both result in no redirect, they differ in meaning. Expired shortlinks are intended to signal that their lifespan is ending clearly. Disabled shortlinks, on the other hand, have been deliberately taken out of service and may be in a temporary state.

Two HTTP status codes represent this semantic separation:

• 410 Gone – for expired short links. This status indicates that the resource is permanently unavailable and will not become available again.
• 404 Not Found – for disabled shortlinks. Although the shortlink exists, its redirect is intentionally not carried out. The 404 status code indicates a temporary or permanent error.

This distinction provides users, API clients, and monitoring systems with more precise feedback on the shortlink’s state. Understandably, a shortlink is not accessible due to a natural process or a manual decision.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-basic-login-solution-part-2/Thu, 18 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-basic-login-solution-part-2/What has happened so far? In the first part of “Basic Login Solution”, the foundation for a deliberately simple yet structurally clean admin login was laid. The starting point was the realisation that even a technically lean URL shortener requires a clear separation between public-facing functions and administrative operations. The goal was not complete user management, but rather a minimal access barrier that integrates seamlessly with the existing Java and Vaadin architectures.<![CDATA[

What has happened so far?

In the first part of “Basic Login Solution”, the foundation for a deliberately simple yet structurally clean admin login was laid. The starting point was the realisation that even a technically lean URL shortener requires a clear separation between public-facing functions and administrative operations. The goal was not complete user management, but rather a minimal access barrier that integrates seamlessly with the existing Java and Vaadin architectures.

The central component of this solution is a lean, file-based configuration inauth.properties. With just two parameters – an activation switch and a password – the login can be fully controlled. The associatedLoginConfigInitializer ensures that this configuration is read when the servlet container starts and remains consistently available to the application. This clearly defines whether and how the protection mechanism takes effect even before any UI is rendered.

Based on this, an independent login view was introduced that does not require MainLayout. It serves as a clear entry point into the administrative area and separates the login context from the rest of the UI, both visually and technically. The implementation focuses on a simplified user experience: a password field, a clear call to action, and clear feedback on success or failure. At the same time, an important architectural principle of today is already evident here: security logic and UI interaction remain neatly separated.

After completing Part 1, a functional login is available, but no full access control is yet in place. Without additional measures, unauthenticated users could continue to access administrative views directly, for example, via deep links or saved URLs. This is precisely where the second part comes in.

InPart 2, login is effectively enforced for the first time: via central route protection in MainLayout, consistent session management with SessionAuth, and a clean logout mechanism. This transforms an isolated login screen into a complete, end-to-end authentication flow that reliably protects the application’s administrative area.

The source code for this article can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-09

Route Protection & Access Logic

With the introduction of admin login, it is not enough to provide only a login page. The decisive factor in the protection’s effectiveness is that all administrative views are consistently protected and accessible only to authenticated users. This is precisely where Route Protection comes in: in conjunction with LoginConfig and SessionAuth, it ensures the application clearly distinguishes between logged-in and non-logged-in users.

Instead of securing each view individually, the implementation uses a single central location: theMainLayout. Since all relevant management views use this layout, it is a natural anchor point for bundling the access logic. As soon as a view with this layout is loaded, the MainLayout can check whether login is enabled and whether the current user is already authenticated. Only when these conditions are met is access to the target view granted.

There are several benefits to implementing route protection within the layout. On the one hand, the code in individual views remains lean because they do not have to perform security checks themselves. On the other hand, a uniform logic is implemented that applies equally to all protected areas. If the mechanism is later expanded or adapted, a change at a central location can affect behaviour across the entire application.

From a functional perspective, route protection follows a simple process: when a navigation enters a view that uses MainLayout, it checks whether login is globally enabled before rendering. If this is not the case, the application behaves as before and still allows direct access to all pages. If login is enabled, the logic checks whether the current session is marked as authenticated. If not, the user will be automatically redirected to the login page.

An exception is the login view itself. It must always be accessible without prior authentication, as it enables access to the protected area. If they were also subject to route protection, an infinite redirect loop would result. The implementation explicitly handles this special case by excluding the login route from the check.

In practice, this mechanism results in transparent, predictable behaviour: non-logged-in users who open a deep link directly into the admin UI are automatically redirected to the login page. After a successful login, you will be taken to the desired administration page and can navigate the protected area without further interruptions. At the same time, the option to altogether turn off the login at the configuration level is retained – in this case, the application behaves as if route protection never existed.

Implementation in MainLayout

The technical implementation of route protection is anchored in theMainLayout. This not only serves as a visual framework for the administration interface but also assumes a central control function for all navigation leading to the protected area via the BeforeEnterObserver interface.

First, the layout is extended so that it can participate in navigation and, at the same time, receive logging functionality:

javaCopy

extendsAppLayoutimplementsBeforeEnterObserver,HasLogger{

The implementation ofBeforeEnterObserver enables the MainLayout to perform a check before each navigation to a View that uses it. At the same time,HasLogger provides a convenient logging API to make decisions and states in the log traceable.

The actual route protection isbundled in thebeforeEnter method:

javaCopy

@OverridepublicvoidbeforeEnter(BeforeEnterEventevent){Ifloginisgloballyturnedoff,donotprotectanyroutes.if(!LoginConfig.isLoginEnabled()){return;}logger().info("beforeEnter target={} authenticated={}",event.getNavigationTarget().getSimpleName(),SessionAuth.isAuthenticated());Theloginviewitselfmustneverbeprotected,otherwisewecreatealoopif(event.getNavigationTarget().equals(LoginView.class)){return;}if(!SessionAuth.isAuthenticated()){logger().info("beforeEnter.. isAuthenticated()==false - reroute to LoginView");event.rerouteTo(LoginView.class);}}

The logic follows the process described above exactly. First, it is checked whether the login is active per the configuration. Iflogin.enabled is set to false in the auth.properties file, the method returns immediately without any restrictions. In this mode, the application behaves as it did before Route Protection was introduced.

If login is enabled, the next step is to check the current navigation destination and verify whether the user is already authenticated—a short log entry records which view to navigate to and the current authentication status. If the target is theLoginView, the check is aborted – the login page always remains accessible, regardless of whether the session is already logged in or not.

For all other views, if the session is not marked as logged in bySessionAuth.isAuthenticated(), the user will be transparently redirected to the login page. The original target view is not rendered, so no administrative functions are visible without authentication.

Logout button in the header

Closely linked to route protection is the ability to clean end an existing session. This functionality is also implemented in theMainLayout and appears to the user as a logout button in the header. The creation of this button is linked to the configuration:

javaCopy

HorizontalLayoutheaderRow;if(LoginConfig.isLoginEnabled()){varlogoutButton=newButton("Logout",_->{SessionAuth.clearAuthentication();UI.getCurrent().getPage().setLocation("/"+LoginView.PATH);});logoutButton.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE);headerRow=newHorizontalLayout(toggle,appTitle,newSpan(),storeIndicator,logoutButton);}else{headerRow=newHorizontalLayout(toggle,appTitle,newSpan(),storeIndicator);}

If the login is deactivated, no logout button is displayed because there is no state to terminate. If the login is active, MainLayout adds a simple inline logout button to the header. Clicking it calls SessionAuth.clearAuthentication(), removes the authentication attribute from the VaadinSession, and closes the session. The browser is then explicitlyredirected to the login page via setLocation.

The combination ofthe beforeEnter check and the logout button creates consistent access logic: Users are directed to the login page when they first access the protected area, can move freely within the admin UI after successful login, and can end their session at any time in a visible, controlled manner.

In the next chapter, we will focus on theSessionAuth class, which encapsulates the login status in the session and provides a centralised access point.

Logout function in the header

A login mechanism is only complete if it also knows a clear way back. In practice, this means that anyone who logs in to the admin interface must be able to end their session just as consciously. The introduction of a logout function is therefore not only a technical addition but also an essential signal to users that access to the administrative area is considered a sensitive context.

In the URL shortener, this functionality is shown as a slim logout button in the header. It is visible only if login protection is enabled in the configuration and the application is in “protected mode”; in all cases where “login.enabled=false", the UI does not display this button because there is no session to log off. This deliberately keeps the interface tidy in simple development or demo scenarios.

From the user’s perspective, the logout button is inconspicuously integrated into the existing header. It does not appear as a prominent CTA, but it is always available in the admin area. The naming is deliberately kept clear: “Logout” leaves no room for interpretation and signals that the current authorisation context is ended here. In environments where multiple people work one after the other using the same browser and admin interface, this clarity is essential.

Technically, clicking the logout button performs two tasks: first, the authentication status in the current VaadinSession is reset; second, the browser is redirected to the login page. This ensures that the administrative views do not remain open in the browser; subsequent access still follows the login flow. Even an accidentally open tab loses its authorisation as soon as the user logs out.

The implementation adheres closely to the rest of the login architecture. The button does not check itself whether a session is authenticated; instead, it delegates this to the small helper class SessionAuth, which is also used elsewhere, such as for route protection. This keeps the logout consistent: the same abstraction that determines whether a session is considered logged in is also responsible for deleting this status.

From a UX perspective, the logout function also helps structure users’ mental model. As long as they are logged in, they are in an active administration context where changes to shortlinks and configurations are expected. When they log out, they deliberately revert to a neutral role, in which they use the generated URLs at most from the user’s perspective. This separation is particularly helpful in heterogeneous teams where not all participants need administrative rights simultaneously.

Implementation in MainLayout

The logout function is centrally integrated intothe MainLayout. The header is built there, and the logout button can be added alongside the app title and the store indicator. Whether this button is visible depends directly on the login configuration:

javaCopy

HorizontalLayoutheaderRow;if(LoginConfig.isLoginEnabled()){varlogoutButton=newButton("Logout",_->{SessionAuth.clearAuthentication();UI.getCurrent().getPage().setLocation("/"+LoginView.PATH);});logoutButton.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE);headerRow=newHorizontalLayout(toggle,appTitle,newSpan(),storeIndicator,logoutButton);}else{headerRow=newHorizontalLayout(toggle,appTitle,newSpan(),storeIndicator);}

In activated mode, a “Logout” button is created and added to the header’s HorizontalLayout. The look is based on the rest of the UI and uses theLUMO_TERTIARY_INLINE variant, so the button appears discreet and blends visually with the header.

The actual logoff logic is in the click listener: First,SessionAuth.clearAuthentication() is called to reset the current session’s authentication state. The browser is then explicitly redirected to the login view path. This means that the user is not just taken to a “blank” page after logging out, but is clearly recognisable back to the entry point of the admin area.

If the login function is globally deactivated, the button is omitted. TheheaderRow layout then consists only of toggle, title, spacer andStoreIndicator. This deliberately keeps the UI slim in scenarios without login protection and does not display a non-working logout button.

SessionAuth – Session state management

The SessionAuthhelper classincludes all accesses to the authentication state within theVaadinSession. It is used by both the route protection and the logout button and thus represents the central abstraction layer for login decisions:

javaCopy

packagecom.svenruppert.urlshortener.ui.vaadin.security;importcom.svenruppert.dependencies.core.logger.HasLogger;importcom.vaadin.flow.server.VaadinSession;import staticcom.svenruppert.dependencies.core.logger.HasLogger.staticLogger;import staticjava.lang.Boolean.TRUE;publicfinalclassSessionAuthimplementsHasLogger{privatestaticfinalStringATTR_AUTH="authenticated";privateSessionAuth(){}publicstaticbooleanisAuthenticated(){VaadinSessionsession=VaadinSession.getCurrent();if(session==null)returnfalse;varattribute=session.getAttribute(ATTR_AUTH);staticLogger().info("isAuthenticated.. {}",attribute);returnTRUE.equals(attribute);}publicstaticvoidmarkAuthenticated(){staticLogger().info("markAuthenticated.. ");VaadinSessionsession=VaadinSession.getCurrent();if(session!=null){session.setAttribute(ATTR_AUTH,TRUE);}}publicstaticvoidclearAuthentication(){staticLogger().info("clearAuthentication.. ");VaadinSessionsession=VaadinSession.getCurrent();if(session!=null){session.setAttribute(ATTR_AUTH,null);session.close();}}}

ThemarkAuthenticated() method is called after a successful login and setsthe simple attributeauthenticatedtoTRUE in the currentVaadinSession. WithisAuthenticated(), this state can be queried again later – for example, in the route protection of theMainLayout. Both methods use logging to quickly determine when a session was logged in or logged out, in the event of an error, or when analysing user behaviour.

For the logout,clearAuthentication() is crucial. It removes the attribute from the session and also closes theVaadinSession. This will also discard other session-related data, and an accidentally opened browser tab will lose its permissions. The next time it is accessed, a new, unauthenticated session is established and guided through the login flow.

This tight integration ofMainLayout andSessionAuth creates a robust, yet manageable, logout implementation. The header provides a clearly visible exit from the admin context, and the session logic ensures this exit is implemented consistently from a technical standpoint.

In the next chapter, we will take a closer look at the SessionAuth class and examine its role in interacting with route protection and the login view within the overall application flow.

SessionAuth: Session-based authentication in the overall flow

Now that login protection, the login page, and the logout function have been introduced, a central question remains: Where is it actually recorded whether a user is authenticated? The answer lies in the small but crucial helper class SessionAuth, which anchors the authentication state in VaadinSession.

At its core,SessionAuth fulfils three tasks. It can first check whether a current session is already registered. Second, after successful login, it can mark the session as authenticated. And thirdly, it can remove the authentication status when logging out and close the session. These three operations form the basis of the login view, route protection in the MainLayout, and the logout button in the header, all of which access a standard truth value rather than maintaining their own state.

The chosen approach leverages the fact that Vaadin maintains a separate VaadinSession for each user. This session exists on the server side and is therefore less susceptible to client-side manipulation than, for example, a simple browser flag. Bysetting a dedicated attribute – such as“authenticated” – in this session,SessionAuth clearly links the login state to the current browser session. If the same user opens a new browser window, a new session is created; the user is not logged in again.

In the interaction between components, this results in a precise flow: if a user reaches the login page and enters a correct password, LoginView calls SessionAuth.markAuthenticated() after a successful check. It then navigates to the administrative overview. As soon as the user opens additional views from there, the MainLayout checks whether the session is still considered logged in, as part of the route protection, usingSessionAuth.isAuthenticated(). If this is the case, navigation is allowed. Otherwise, the request will be redirected to the login page.

Finally, if the user clicks the logout button in the header, SessionAuth.clearAuthentication() removes the authentication attribute from the session and closes VaadinSession. For the server, this session is over. The following request establishes a new session, which is again considered unauthenticated from the application’s perspective, so Route Protection redirects the request back to the login view.

This session approach fits well with the project’s objective: it is deliberately kept simple, requires no additional infrastructure and is easy to understand. At the same time, it meets the requirements of a minimalist admin login, which is less about highly secured, distributed authentication procedures and more about a clear separation between “logged in” and “not logged in” within a running browser session.

In the rest of this chapter, we will go through the implementation of the three methodsisAuthenticated,markAuthenticated andclearAuthentication and look at how they interact at the different points in the code – Login-View, MainLayout and Logout-Button.

The implementation of SessionAuth in detail

TheSessionAuth class is deliberately kept compact. It encapsulates access to theVaadinSession and provides three static methods, each of which fulfils a clearly defined task:

javaCopy

packagecom.svenruppert.urlshortener.ui.vaadin.security;importcom.svenruppert.dependencies.core.logger.HasLogger;importcom.vaadin.flow.server.VaadinSession;import staticcom.svenruppert.dependencies.core.logger.HasLogger.staticLogger;import staticjava.lang.Boolean.TRUE;publicfinalclassSessionAuthimplementsHasLogger{privatestaticfinalStringATTR_AUTH="authenticated";privateSessionAuth(){}publicstaticbooleanisAuthenticated(){VaadinSessionsession=VaadinSession.getCurrent();if(session==null)returnfalse;varattribute=session.getAttribute(ATTR_AUTH);staticLogger().info("isAuthenticated.. {}",attribute);returnTRUE.equals(attribute);}publicstaticvoidmarkAuthenticated(){staticLogger().info("markAuthenticated.. ");VaadinSessionsession=VaadinSession.getCurrent();if(session!=null){session.setAttribute(ATTR_AUTH,TRUE);}}publicstaticvoidclearAuthentication(){staticLogger().info("clearAuthentication.. ");VaadinSessionsession=VaadinSession.getCurrent();if(session!=null){session.setAttribute(ATTR_AUTH,null);session.close();}}}

isAuthenticated() – Check if a session is logged in

TheisAuthenticated() method is the primary read-only interface. It is used, among other things, in theMainLayout to decide whether navigation should be allowed or redirected to the login view as part of route protection. Internally, she first asks about the currentVaadinSession. If no session exists (for example, in the very early stages of a request), it returns false immediately.

If a session exists, the “authenticated” attribute is read. It is a simple object value set to “Boolean.TRUE” upon successful login. The method compares this value toTRUE and returnstrue orfalse accordingly. The additional log entry helps track how often and with what results this check is invoked in the running system.

markAuthenticated() – Mark session as logged in

After a successful password check in theLoginView,markAuthenticated() is called. The method fetches the current VaadinSession and sets the “authenticated" attributetoTRUE. This updates the session state so that subsequent calls to isAuthenticated() return true for that session.

Here, too, a log entry ensures that the registration time remains traceable. If unexpected conditions occur in the company – for example, because users report that they are “suddenly logged out again” – the logs can be used to understand better when sessions were marked or discarded.

clearAuthentication() – log out and close session

The third method, clearAuthentication(), is the counterpart to login. The logout button in the MainLayout invokes it and performs two tasks simultaneously. First, it removes the"authenticated" attribute from the current session by setting it tonull. This means isAuthenticated() will return false for this session from this point on.

In the second step, “session.close()" is called. This invalidates the entireVaadinSession, including any other attributes that may have been set. This measure ensures that other session-related information is not forwarded and that a new request actually starts with a fresh session.

In combination with the explicit navigation back to the login page, this results in a clean logout flow: the session loses its authorisation, the user leaves the admin context, and must authenticate again to continue.

Interaction with Login View and MainLayout

In the overall flow of the application,SessionAuth is used in several places:

• In theLoginView, theattemptLogin()method callsSessionAuth.markAuthenticated()after a successful password comparisonand then navigates to the overview page.
• Inthe MainLayout, the beforeEnter() implementation uses SessionAuth.isAuthenticated() to intercept unauthorised access and, if necessary, redirect to the login view.
• The logout button in the header callsSessionAuth.clearAuthentication() and then redirects the user out of the administration context to the login page.

In this way,SessionAuth centralises all access to the authentication status in a single place. Changes to the session model – such as a different attribute name or additional metadata – are made here and are then consistently available to all consuming components.

Conclusion & Outlook

With the introduction of a simple, configurable admin login, the application gains a clearly defined protection mechanism for all administrative functions without the complexity of a full-fledged authentication framework. The solution is deliberately tailored to the project’s characteristics: lightweight, comprehensible, and implemented using pure Java/Vaadin architecture. At the same time, it covers the most essential requirements for minimal access control – from password requests to route protection to a clean logout mechanism.

The overall system consists of a few clearly separated building blocks: a central configuration file, the LoginConfig for evaluating these values, a simple login view for user interaction, route protection in the MainLayout, and the SessionAuth class for managing session state. These components interlock like gears, creating a process that remains technically sound and intuitive for users.

At the same time, the implementation is deliberately extensible. The current approach stores the password in plain text in the configuration file and uses it, unchanged, as a byte sequence within the application. This is sufficient for development and internal scenarios, but it is clear that it is not suitable for higher security requirements. However, the architecture leaves room for the following points of expansion:

• Password hashing: The stored passwords are hashed using a hash function such as SHA-256 or bcrypt, so they are not stored in plaintext on the server.
• Multiple users or roles: The structure could be expanded to support various administrators with individual passwords.
• Time-limited sessions: An automatic logout due to inactivity would further enhance security.
• Two-Factor Authentication (2FA): For more demanding environments, a second level of security – for example via TOTP – could be added.
• External identity providers: The application could be connected to OAuth 2.0/OpenID Connect in the long term, provided it fits the application scenario.

However, the strength of the solution lies precisely in its not attempting to anticipate these aspects. It provides a pragmatic, ready-to-use foundation that reliably protects the admin area from unintended access without unnecessarily complicating deployment or development.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-basic-login-solution-part-1/Wed, 17 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-basic-login-solution-part-1/Introduction The administration interface of a URL shortener is a sensitive area where short links can be changed, removed, or assigned expiration dates. Although the system is often operated on an internal server or in a private environment, protecting this management interface remains a fundamental security concern. Accidental access by unauthorised users can not only lead to incorrect forwarding or data loss, but also undermine trust in the overall system.<![CDATA[

Introduction

The administration interface of a URL shortener is a sensitive area where short links can be changed, removed, or assigned expiration dates. Although the system is often operated on an internal server or in a private environment, protecting this management interface remains a fundamental security concern. Accidental access by unauthorised users can not only lead to incorrect forwarding or data loss, but also undermine trust in the overall system.

With the introduction of a configurable login mechanism, precise access control is introduced, deliberately separating access to management functions from the rest of the system. The login serves as a lightweight security measure that does not require external dependencies, frameworks or time-consuming user management. It is precisely this simplicity – a single password, a simple configuration file and a centred login screen – that makes the solution particularly attractive for small deployments or personal projects.

The source code for this article can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-09

Objectives of the implementation

The introduction of a simple, configurable login mechanism aims to achieve several clearly defined goals that improve both the security and usability of the URL shortener. The focus is not on establishing a complex user and role model, but on creating a pragmatic protective layer that reliably controls administrative access without unnecessarily increasing development or operating costs.

First, the primary objective is to secure administrative functions. The management interface allows you to create, edit, and remove short links, as well as set or clean expiration dates. These functions must be protected against unauthorised access, particularly if the URL shortener is not operated exclusively within the closed network. A simple login prevents accidental or curious access from causing damage.

Another goal is to minimise the barrier to entry. The solution should work without additional frameworks, external identity providers, or complex configuration. The implementation is deliberately based on the project’s philosophy: Everything remains manageable, lightweight, and understandable. With a simple auth.properties file, the system can be turned on or off at will, and a single password suffices for access control.

In addition, the login mechanism is designed to ensure a predictable and consistent user experience. These include a well-designed login page, an immediate redirect on unauthenticated access to protected areas, and a transparent logout process that cleanly ends the session. All these aspects ensure that the login fits naturally into the existing UI and is still clearly perceived as a security measure.

Finally, the login also serves as a foundation for potential future expansions. Although the current implementation is deliberately minimalist, it provides the structural prerequisites to respond to stronger security requirements if necessary—for example, by adding hashing mechanisms, time-limited session tokens, or server-side password rotation. This leaves the system open for further development without losing its current simplicity.

Configurable login viaauth.properties

The new login mechanism is based on a deliberately simple configuration file that enables centralised control of the application’s authentication behaviour. This file is namedauth.properties and is located in the application’s resource directory. By using them, the login system can not only be conveniently activated or deactivated, but also quickly adapted to different deployment scenarios.

The focus is on two configuration keys:login.enabled andlogin.password. While the first parameter checks whether the login system is active, the second parameter specifies the required access password. These values are automatically read when the application starts and henceforth determine the behaviour of all protected areas. It is precisely this mechanism that enables turning off login on short notice or changing the password without modifying the source code again.

The configuration is read in by the LoginConfigInitializer, which is executed when the servlet container starts. It checks whether the file exists, loads its contents and passes it to the central classLoginConfig, which manages these values and makes them available for later access. ``

This class is responsible for correctly configuring the application’s login behaviour at startup. This ensures that both the login page and the route protection can access a uniform configuration basis at all times.

After loading the file, theLoginConfig class assumes responsibility for persistently storing the values and performing password comparisons.

This form of configuration control not only simplifies operation but also supports different operating modes. For example, a fully disabled login system is suitable for purely internal development environments, while the activated mode protects against unauthorised changes in production or publicly available environments. Switching between the two modes is kept as low-threshold as possible – a simple value in a text file is enough.

Although this mechanism provides basic protection, it isnot designed to be a highly secure authentication solution. Instead, the goal is to establish a minimum level of security to protect the administrative area from accidental access or unauthorised use. For productive environments with increased security requirements, additional measures such as password hashing, multi-factor authentication, or integration into established identity services would be necessary.

LoginConfig & Initialization

Central control of login behaviour is based on two closely interlinked components: the LoginConfig class itself and its upstream initialiser,LoginConfigInitializer. Together, they ensure that the configuration from theauth.properties file is correctly read, interpreted, and made available to the application runtime.

The first focus is on theLoginConfig class. It provides a minimalist yet coherent foundation for the login system. The approach is intentionally simple: there is no user base, roles, or profiles, just a single password that serves as an access threshold. The class manages this password and the information on whether the login should be active. The structure remains manageable to keep the entry barrier for administrators and developers low.

An essential detail is the division into two phases: first, when the application starts, the LoginConfigInitializer checks which settings are stored in the configuration file. These values are then passed to the staticLoginConfig.initialise() method, which populates the appropriate fields and makes them available to the rest of the application.

This initialisation process occurs entirely when the servlet container starts. This ensures that all views loaded later, especially the route protection and the login page, have access to a consistent and fully initialised configuration. This avoids error conditions that could arise from missing or delayed loading of configuration values.

In the following, we will take a closer look at both components and refer to the source code for clarity.

LoginConfig – central configuration class

The core is theLoginConfig class, which stores the login switch and the expected password as byte data. It is declaredfinal and has a private constructor, so it is used exclusively through its static methods:

javaCopy

/** * Central configuration for the simple admin login. * Reads its values from auth.properties via LoginConfigInitializer. */publicfinalclassLoginConfig{privatestaticvolatilebooleanloginEnabled;privatestaticvolatilebyte[]expectedPasswordBytes;privateLoginConfig(){}/** * Initialises the login configuration. * * @param enabled whether the login mechanism should be enforced * @param password the raw password read from configuration, may be {@code null} */publicstaticvoidinitialise(booleanenabled,Stringpassword){loginEnabled=enabled;if(!enabled||password==null||password.isBlank()){expectedPasswordBytes=null;return;}expectedPasswordBytes=password.getBytes(StandardCharsets.UTF_8);}/** * @return {@code true} if login protection is enabled at all */publicstaticbooleanisLoginEnabled(){returnloginEnabled;}/** * @return {@code true} if login is enabled and a usable password has been configured */publicstaticbooleanisLoginConfigured(){returnloginEnabled&&expectedPasswordBytes!=null&&expectedPasswordBytes.length>0;}/** * Compares the entered password with the configured one using constant-time comparison. */publicstaticbooleanmatches(char[]enteredPassword){if(!isLoginConfigured()||enteredPassword==null){returnfalse;}byte[]entered=newString(enteredPassword).getBytes(StandardCharsets.UTF_8);booleanresult=MessageDigest.isEqual(expectedPasswordBytes,entered);Best-effortclean-upArrays.fill(entered,(byte)0);returnresult;}}

Theinitialise method is called only when the initialiser starts and determines whether login is enabled (loginEnabled) and whether a valid password is present. Invalid or empty passwords are consistently discarded; isLoginConfigured() only returns true if there is a usable configuration.

For the actual password comparison,matches(char[] enteredPassword) serves as the central function. It takes the entered password as a char[], converts it to a byte array in UTF-8 format, and compares it with the expected byte sequence usingMessageDigest.isEqual. This enables constant-time comparison, making simple timing attacks more difficult. The temporary byte array is then overwritten using Arrays.fill to remove the sensitive data from memory, at least as effectively as possible.

LoginConfigInitializer – Load configuration on startup

For LoginConfig to work with meaningful values, the configuration file must be read when the servlet container starts. This task is performed by theLoginConfigInitializer class, whichis registered as @WebListener andimplements the ServletContextListener interface :

javaCopy

@WebListenerpublicclassLoginConfigInitializerimplementsServletContextListener,HasLogger{privatestaticfinalStringPROPERTIES_PATH="auth.properties";@OverridepublicvoidcontextInitialized(ServletContextEventsce){logger().info("Initialising LoginConfig from {}",PROPERTIES_PATH);Propertiesprops=newProperties();try(InputStreamin=getClass().getClassLoader().getResourceAsStream(PROPERTIES_PATH)){if(in==null){logger().warn("No {} found on classpath. Login will be disabled.",PROPERTIES_PATH);LoginConfig.initialise(false,null);return;}props.load(in);StringenabledRaw=props.getProperty("login.enabled","true").trim();booleanenabled=Boolean.parseBoolean(enabledRaw);Stringpassword=props.getProperty("login.password");LoginConfig.initialise(enabled,password);if(!enabled){logger().info("Login explicitly disabled via login.enabled=false");}elseif(LoginConfig.isLoginConfigured()){logger().info("LoginConfig initialised successfully from {}",PROPERTIES_PATH);}else{logger().warn("login.enabled=true but no usable password configured. "+"Login will effectively be disabled.");}}catch(IOExceptione){logger().error("Failed to load "+PROPERTIES_PATH+". Login will be disabled.",e);LoginConfig.initialise(false,null);}}@OverridepublicvoidcontextDestroyed(ServletContextEventsce){Nothingtocleanup}}

The initialiser follows a straightforward process: first, it attempts to load the auth.properties file from the classpath. If this is not possible, the login is explicitly deactivated, and a warning log is written. If successful, the configuration values “login.enabled" and “login.password" are read, converted to appropriate data types, and forwarded to LoginConfig.initialise. Finally, depending on the configuration, additional log output is generated, providing quick information about the active mode during operation.

This separation of responsibilities creates a well-traceable initialisation path: LoginConfigInitializer handles loading and interpreting the configuration file. At the same time, LoginConfig encapsulates the actual login logic and is later used by other components, such as the login view or route protection.

The new login page

With the introduction of admin login, the application gains its own entry page that deliberately distinguishes itself from the rest of the UI. The new login page serves as the boundary between public functions, such as link forwarding, and sensitive administrative functions in the backend. The goal was to create a reduced, highly focused layout that integrates seamlessly with the application’s visual design while remaining clearly recognisable as a security barrier.

Instead of being embedded in the existing navigation layout, the login is rendered as a standalone view withoutMainLayout. Users will not see any page navigation, drawer or admin functions until they have successfully authenticated. The page fills the entire browser window; its contents are centred vertically and horizontally. This creates the impression of a classic login screen with a single task: requesting the password for the administrative area.

At the centre is a compact login panel comprising a headline, a short explanatory text, a password field, and a login button. The headline clearly conveys that this is admin access, while the accompanying text explains why a password is required and what area it protects. This improves transparency and reduces queries, especially when multiple users use the system.

The password field is configured to automatically receive focus when the page is loaded. Users can therefore type directly without first clicking with the mouse. In addition, a clear button lets you remove an accidentally entered string with a single click. Displaying the password in plain text is deliberately avoided to reduce the risk of shoulder-surfing, especially in shared work environments.

The input behaviour is also designed to ensure a smooth process. Authentication can be done either by clicking on the login button or by pressing Enter. If the login fails, the view marks the password field as invalid and displays a clear error message indicating that the entered password is incorrect. This preserves context and allows the user to start a second attempt immediately without reloading the page.

Implementation of the LoginView

The technical implementation of the described login page is handled by theLoginView class. It is registered as a separate route and deliberately dispenses with a layout to enable a focused, full-surface login screen:``

javaCopy

@Route(LoginView.PATH)// No layout = no navigation or drawer visible@PageTitle("Admin Login | URL Shortener")publicclassLoginViewextendsVerticalLayoutimplementsBeforeEnterObserver{publicstaticfinalStringPATH="login";privatefinalPasswordFieldpasswordField=newPasswordField("Password");privatefinalButtonloginButton=newButton("Login");publicLoginView(){setSizeFull();setAlignItems(Alignment.CENTER);setJustifyContentMode(JustifyContentMode.CENTER);configureForm();buildLayout();}

It is already evident from the declaration that LoginView is not embedded inMainLayout. Because the route annotation lacks a layout, the page is displayed in isolation; the layout properties in the constructor handle complete vertical and horizontal centring. The two central UI components – password field and login button – are kept as fields so that theycan be further configured in thehelper methods configureForm()andbuildLayout().

The configuration of the form is deliberately designed for a lean but fluid user experience:

javaCopy

privatevoidconfigureForm(){passwordField.setAutofocus(true);passwordField.setWidth("300px");passwordField.setClearButtonVisible(true);passwordField.setRevealButtonVisible(false);passwordField.setInvalid(false);loginButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY);loginButton.setWidth("300px");loginButton.addClickListener(_->attemptLogin());passwordField.addKeyDownListener(event->{if("Enter".equalsIgnoreCase(event.getKey().getKeys().getFirst())){attemptLogin();}});passwordField.addValueChangeListener(_->passwordField.setInvalid(false));}

The password field automatically gains focus when the page loads and is set to a fixed width, so that the input and button visually form a single unit. The Clear button lets you quickly delete an incorrect entry; the password is displayed in plain text by default. The login button is marked as the primary button and, when pressed, triggers the attemptLogin() method. The value change listener ensures that a previously set error state is restored when typing again.

The structure of the panel that appears in the middle of the screen is defined inbuildLayout():

javaCopy

privatevoidbuildLayout(){H2title=newH2("Admin Login");Paragraphsubtitle=newParagraph("Please enter the administrator password to access the management interface.");VerticalLayoutformLayout=newVerticalLayout(title,subtitle,passwordField,loginButton);formLayout.setSpacing(true);formLayout.setPadding(true);formLayout.setAlignItems(Alignment.CENTER);add(formLayout);}

The combination of the heading, explanatory paragraph, password field, and button forms exactly matches the compact login panel described in the previous section. Centring the innerVerticalLayout creates a clear focus on the input area, without visual distractions from other UI elements.

The actual login process is encapsulated inattemptLogin(). This is where configuration and user input are merged:

javaCopy

privatevoidattemptLogin(){if(!LoginConfig.isLoginEnabled()){Notification.show("Login is currently disabled. Please check the server configuration."3000,Notification.Position.MIDDLE);UI.getCurrent().navigate(OverviewView.PATH);return;}char[]input=passwordField.getValue()!=null?passwordField.getValue().toCharArray():newchar[0];if(!LoginConfig.isLoginConfigured()){Notification.show("Login is not configured. Please verify that the configuration file has been loaded.",3000,Notification.Position.MIDDLE);return;}booleanauthenticated=LoginConfig.matches(input);if(authenticated){SessionAuth.markAuthenticated();UI.getCurrent().navigate(OverviewView.PATH);}else{passwordField.setErrorMessage("Incorrect password");passwordField.setInvalid(true);}}

First, it is checked whether the login is enabled in the configuration. If this is not the case, the user receives clear feedback via notification and is redirected directly to the overview page. The next step is to check whether the login has been configured correctly. Only when both conditions are met is the entered password passed toLoginConfig.matches. If successful, the session is marked as authenticated viaSessionAuth.markAuthenticated() and forwarded to the overview. In the event of an error, the view marks the password field as invalid and displays a clear error message – the user remains on the login page and can adjust the input.

Finally, the implementation of theBeforeEnterObserver ensures that the login page itself only remains visible when it makes sense:

javaCopy

@OverridepublicvoidbeforeEnter(BeforeEnterEventevent){Ifloginisdisabled,skiptheloginpageentirelyif(!LoginConfig.isLoginEnabled()){event.forwardTo(OverviewView.PATH);return;}Ifalreadyauthenticated,alsoskiptheloginpageif(SessionAuth.isAuthenticated()){event.forwardTo(OverviewView.PATH);}}Thispreventsauthenticatedusersfrombeingredirectedtotheloginpageagainandensures

This prevents authenticated users from being redirected to the login page again and ensures that a globally deactivated login system does not unnecessarily display a redundant password dialogue.

Another aspect is the interaction with the configuration. If login in the auth.properties file is disabled or not configured meaningfully, the view displays clear instructions. In one case, it indicates that login is currently disabled and redirects directly to the overview page. In the other case, it suggests that a valid configuration could not be loaded. The page thus serves not only as an input form, but also as a diagnostic point where misconfigurations become visible at an early stage.

Overall, the new login page ensures that access to the admin interface is structured, predictable and visually clearly distinguished from the rest of the system. It combines a deliberately simple interaction with a comprehensible security concept, thereby laying the foundation for the other building blocks of the login flow, particularly route protection and session management.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-mass-grid-operations-part-2/Tue, 16 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-mass-grid-operations-part-2/What has happened so far… In the previous part, the URL shortener overview was significantly expanded. The starting point was the realisation that the last UI was heavily optimised for single operations and thus quickly reached its limits as soon as larger volumes of shortlinks needed to be managed. To resolve this bottleneck, the grid was consistently switched to multiple selection, creating the technical basis for actual mass operations.<![CDATA[

What has happened so far…

In the previous part, the URL shortener overview was significantly expanded. The starting point was the realisation that the last UI was heavily optimised for single operations and thus quickly reached its limits as soon as larger volumes of shortlinks needed to be managed. To resolve this bottleneck, the grid was consistently switched to multiple selection, creating the technical basis for actual mass operations.

Based on this, a context-dependent bulk action bar was created, visible only when a selection is present. It serves as a central control for simultaneous multi-entry actions and seamlessly integrates them into the existing search, filtering, and grid interaction workflow. The decisive factor was not the mere addition of new buttons, but the clean coupling of UI state, selection and action permission.

The first concrete mass operation was the bulk delete. This function exemplified how efficiency and security can be combined: through explicit confirmation dialogues, meaningful previews of the affected entries, consistent feedback, and clear visual cues for destructive actions. This workflow has been supplemented with keyboard shortcuts that speed up deleting larger quantities without bypassing the control mechanisms.

This completes the transition from an entry-centric interface to a professional management view. The overview view now understands selections as a work context and can execute coordinated actions in a controlled manner. The following sections build on this foundation, examining further mass operations, their semantic differences, and their technical implementation in detail.

The source code for this article can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-08

Bulk Clear Expiry: Reliably and Consistently Remove Expiration Dates

While setting a uniform expiration date is a typical administrative task, the targeted removal of such expiration dates also plays a central role. In many real-world scenarios, temporary restrictions lose their relevance: links should remain valid permanently, test or campaign parameters must be reset, or previously set durations have proven too restrictive. The ability to delete expiration dates for many shortlinks in a single step is, therefore, a crucial building block for effective mass management.

The bulk clear expiry function follows a similar interaction pattern to setting an expiration date. Still, it is fundamentally different in its consistency and semantics: while setting introduces a new value, deleting an expiration date explicitly returns to the “does not automatically expire” state. The architecture must ensure that this state is clearly communicated and implemented correctly.

The removal of the expiration time must be understood explicitly, not as an implicit side effect, but as a deliberate operation that requires its own confirmation. This ensures both the security of user interactions and the traceability of changes.

The focus is on the interaction among dialogue logic, API calls, and the changed semantics of the edit endpoint, which now distinguishes between “set new expiration date” and “delete expiration date”. This makes the interaction between the UI and the backend clearly comprehensible and demonstrates how reliable mass changes can be integrated cleanly.

The entry point of the functionality lies in themethod confirmBulkClearExpirySelected(), which – analogous to the bulk delete – first validates the current selection and then opens a confirmation dialogue:

javaCopy

privatevoidconfirmBulkClearExpirySelected(){varselected=grid.getSelectedItems();if(selected.isEmpty()){Notification.show("No entries selected");return;}Dialogdialog=newDialog();dialog.setHeaderTitle("Remove expiry for "+selected.size()+" short links?");dialog.add(newText("This will remove the expiry date from all selected short links. "+"They will no longer expire automatically."));Buttoncancel=newButton("Cancel",_->dialog.close());Buttonconfirm=newButton("Remove expiry",_->{dialog.close();bulkClearExpiry(selected);});confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY);dialog.getFooter().add(newHorizontalLayout(cancel,confirm));dialog.open();}

The structure of this method makes it clear that removing the expiration time is modelled as a standalone operation that requires confirmation. First, an empty selection is excluded; then a dialogue is created whose title explicitly names both the operation type (“Remove expiry”) and the number of affected entries. The actual content of the dialogue explains the consequence of the action in plain language: The links “will no longer expire automatically”. This means that semantics are conveyed not only implicitly by the UI but also explicitly by language.

The actual implementation of the mass operation is carried out in the outsourced methodbulkClearExpiry(...), which encapsulates the transition from the UI layer to the client API:

xmlCopy

 private void bulkClearExpiry(Set<ShortUrlMapping> selected) { if (selected.isEmpty()) { Notification.show("No entries selected"); return; } int success = 0; int failed = 0; for (var m : selected) { try { boolean ok = urlShortenerClient.edit( m.shortCode(), m.originalUrl() ); if (ok) { success++; } else { failed++; } } catch (IOException ex) { logger().error("Bulk clear expiry failed for {}", m.shortCode(), ex); failed++; } } grid.deselectAll(); safeRefresh(); Notification.show("Cleared expiry: " + success + " • Failed: " + failed); }

The signature of the edit call is particularly remarkable. In contrast to the bulk set expiry, only the shortcode and original URL are transferred; the instant and shortcode are not. This API is explicitly implemented in theURLShortenerClient in such a way that the absence of an expiration date is semantically interpreted as “delete expiry”:

javaCopy

publicbooleanedit(StringshortCode,StringnewUrl)throwsIOException{returnedit(shortCode,newUrl,null);}publicbooleanedit(StringshortCode,StringnewUrl,InstantexpiresAtOrNull)throwsIOException{if(shortCode==null||shortCode.isBlank()){thrownewIllegalArgumentException("shortCode must not be null/blank");}if(newUrl==null||newUrl.isBlank()){thrownewIllegalArgumentException("newUrl must not be null/blank");}finalURIuri=serverBaseAdmin.resolve(PATH_ADMIN_EDIT+"/"+shortCode);finalURLurl=uri.toURL();logger().info("edit - {}",url);// ... Request set-up and dispatch ...}

Finally, on the server side, this semantics is anchored in the store implementations. Both the in-memory store and the EclipseStore-based persistence layer now interpret anullvalue forexpiredAt as a signal to remove the expiration time entirely:

xmlCopy

InMemoryUrlMappingStorevar originalOrNewUrl = url != null ? url : shortUrlMappingOLD.originalUrl();var instant = Optional.ofNullable(expiredAt);var shortUrlMapping = new ShortUrlMapping(shortCode, originalOrNewUrl, shortUrlMappingOLD.createdAt(), instant);store.put(shortUrlMapping.shortCode(), shortUrlMapping);EclipseUrlMappingStorevar originalOrNewUrl = url != null ? url : shortUrlMappingOLD.originalUrl();Optional<Instant> instant = Optional.ofNullable(expiredAt);var shortUrlMapping = new ShortUrlMapping(shortCode, originalOrNewUrl, shortUrlMappingOLD.createdAt(), instant);urlMappings.put(shortUrlMapping.shortCode(), shortUrlMapping);storage.store(dataRoot().shortUrlMappings());

Instead of replacing a missing value with the previous expiration date, expiredAt is now consistently converted to Optional.ofNullable(…). The result is a clear division of semantics: if aninstant is passed, the edit operation sets a new expiration date; ifnull ispassed, an empty optional is created, which corresponds to a complete removal of the existing expiration date. This change enables bulkClearExpiry(…) not only to overwrite expiry information covertly but also to delete it.

The combination of a confirmation dialogue, dedicated mass operation, and precisely defined edit semantics creates a continuous path from the user click to the permanently changed data structure. Bulk-Clear-Expiry is not just a “special case” of the edit function, but a deliberately modelled, independent mass operation with clear, technically and professionally comprehensible meaning.

Keyboard interaction as an accelerator for mass operations

With the introduction of mass grid operations, efficient interaction is becoming increasingly important. While mouse gestures and explicit buttons offer high visibility and explainability, keyboard shortcuts are most effective when users perform recurring actions. In the context of the Overview view, this means that central measurement operations should not only be accessible via the bulk bar but also be able to be triggered directly via the keyboard.

Keyboard interaction fulfils two roles. On the one hand, it serves as a direct accelerator of already established workflows. If you are used to working with keyboard shortcuts, you can trigger searches and deletions without detours via the mouse, reducing noticeable friction, especially with frequent context switches between code, console, and browser. On the other hand, it serves as a semantic condensation of the UI: the presence of a shortcut makes it clear that certain operations are not treated as edge cases but as primary interaction paths.

Today, this idea is being fleshed out in two prominent places. On the one hand, the global search in the overview view has a dedicated shortcut, allowing you to focus directly on the search field. This starts at the interaction’s beginning: a single keyboard shortcut is enough to switch from the web interface’s arbitrary state to a precisely defined search mode. On the other hand, the bulk delete operation is bound to the delete key only when a selection is present in the grid. The keyboard thus serves as an equal trigger for the same confirmation dialogue, which would otherwise be accessible only via the bulk bar.

It is essential for this integration that the keyboard shortcuts are not implemented as hidden, hard-to-discover functions, but as a consistent extension of the existing interaction model. They draw on concepts that have already been introduced – global search, multiple selection, bulk dialogues – and link them to a fluid, operable overall system. If you work exclusively with the mouse, you can use all functions as usual; those who prefer keyboard shortcuts, on the other hand, get much faster access to the same function space.

In the following, we will show how these shortcuts are technically integrated, how they interact with the grid state model, and which protective mechanisms prevent mass operations from being unintentionally triggered.

The central entry point is theaddShortCuts() method, which bundles all global keyboard shortcuts of the Overview view:

javaCopy

privatevoidaddShortCuts(){UIcurrent=UI.getCurrent();current.addShortcutListener(_->{if(globalSearch.isEnabled())globalSearch.focus();},Key.KEY_K,KeyModifier.META);// Bulk delete via Delete-Keycurrent.addShortcutListener(_->{if(!grid.getSelectedItems().isEmpty()){confirmBulkDeleteSelected();}},Key.DELETE);}

The method defines two keyboard interactions, each of which fulfils different semantic roles:

Meta + K– Focus on the global search field

The combination ofKeyModifier.META (⌘ on macOS, Windows logo key on Windows) and the letter K address an established UI pattern in code editors and command palettes. With a single keystroke, the user can go directly to the global search, regardless of which UI element was previously in focus.

The protection provided by the following conditions is remarkable:

javaCopy

if(globalSearch.isEnabled())globalSearch.focus();

This prevents the shortcut from forcing an interaction that would not make sense semantically in the current UI state – for example, if a dialogue is currently open or the search has been deactivated due to active filter modes. The keyboard interaction thus fits respectfully into the overall system.

Delete– Trigger bulk delete if there is a selection

The second central key combination binds the delete key directly to the bulk delete function. The shortcut applies to the entire overview view, but is only activated if there is actually a multiple or single selection:

javaCopy

if(!grid.getSelectedItems().isEmpty()){confirmBulkDeleteSelected();}

This ensures that the delete dialogue appears only when there is an object context; therefore, it is not possible to trigger an operation without a target by randomly pressing Delete. This protection mechanism is essential for mass operations, as the delete key is often used unconsciously and intuitively.

Integration into the UI‑state model

Both shortcuts interfere with the grid’s state logic. The search shortcut uses the activation logic of the globalSearch field, while the delete shortcut is explicitly bound to the grid selection. This creates a natural coupling between keyboard interaction and UI state: Shortcuts are not detached from the UI, but follow the same semantic structure as mouse interaction via the bulk bar.

Why it matters: Acceleration without side effects

While shortcuts are often seen as optional conveniences, they play a crucial role in mass operations. Those who manage large amounts of data frequently switch between different tools – IDE, browser, and terminal. The ability to perform central operations without loss of focus and without visual navigation reduces cognitive load and increases speed. At the same time, the built-in protection queries ensure that no unintentional deletion actions are triggered.

This shows that keyboard interaction is not just a nice extra feature, but an integral part of an efficient, error-robust workflow for mass grid operations, how these shortcuts are integrated into the overview view, how they interact with the state model of the grid, and where protection mechanisms have been deliberately implemented to prevent unintentional mass operations.

A consistent visual language for high-risk mass operations

With the introduction of mass operations, both the functional and visual levels become more complex. Where previously individual actions were placed in isolation in the grid, there are now interconnected chains of interactions that have the potential to have a significant impact on the database – especially in operations such as deleting several shortlinks or changing all expiration dates. This increased impact requires a visual language that creates clarity, clearly highlights risks and helps users to perform each operation consciously and safely.

An essential goal of this visual language is to establish clear semantic distinctions among action types. While harmless operations – such as opening a detailed dialogue – may be unagitated and neutral, risky or destructive actions must be immediately recognisable as such. The introduction of the bulk action bar makes this differentiation imperative: it bundles both low-risk and high-risk actions in a small space, creating visual clarity, a prerequisite for safe usability.

The implementation of these principles follows a strict pattern: by targeting Lumo theme variants, buttons are not only styled but also semantically enhanced. Destructive actions are consistently marked withLUMO_ERROR, whereas accompanying or secondary actions are given the more subtleLUMO_TERTIARY_INLINE. This contrast creates an immediately apparent hierarchy, highlighting potentially dangerous actions without visually cluttering the interface.

The technical basis of this visual language is first evident in the implementation of the bulk action bar. A clear distinction is made here between destructive and non-destructive actions. The corresponding code from theOverviewView clearly shows this:

javaCopy

bulkDeleteBtn.addThemeVariants(LUMO_ERROR,LUMO_TERTIARY_INLINE);bulkSetExpiryBtn.addThemeVariants(LUMO_TERTIARY_INLINE);bulkClearExpiryBtn.addThemeVariants(LUMO_TERTIARY_INLINE);

Only theDelete button bears theLUMO_ERROR mark. This signals at a glance that the action could have irreversible effects. The other two buttons – “Set expiry” and “Clear expiry” – deliberately remain inconspicuous. Their functions are operational, but not destructive, which is whyLUMO_TERTIARY_INLINE offers the appropriate visual restraint here.

This differentiation also continues at the row level in the grid. Each row has two buttons that play completely different roles visually:

sqlCopy

Buttondelete=newButton(newIcon(VaadinIcon.TRASH));delete.addThemeVariants(LUMO_ERROR,ButtonVariant.LUMO_TERTIARY);

The delete button is red, making it immediately recognisable. The corresponding detail button is visually restrained:

kotlinCopy

vardetails=newButton(newIcon(VaadinIcon.SEARCH));details.addThemeVariants(LUMO_TERTIARY_INLINE);

The user immediately recognises which of the two actions is inconsequential and which is critical. This visual semantics is not cosmetic; it reduces misclicks and improves security, especially in fast workflows. This logic continues in the confirmation dialogues. In the case of bulk delete, the confirmation button is marked twice:

javaCopy

Buttonconfirm=newButton("Delete",_->{...});confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY,LUMO_ERROR);

LUMO_PRIMARY marks the main action in the dialogue,LUMO_ERROR immediately makes it clear that this main action is destructive. The red colouring was deliberately chosen to increase attention and prevent accidental confirmations.

However, different semantics apply when removing an expiration date. The process is a mass operation, but not destructive. Therefore, the application explicitly does not use a red marking:

javaCopy

Buttonconfirm=newButton("Remove expiry",_->{...});confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY);

Here, only the primary theme is set – the action is essential, but not dangerous. The UI thus distinguishes between critical and relevant, which is crucial for intuitive usability.

In summary, the consistent use of Lumo themes creates a clear, recognisable hierarchy:

• **LUMO_ERROR** → destructive, potentially irreversible
• **LUMO_PRIMARY** → central affirmative action
• **LUMO_TERTIARY**/**LUMO_TERTIARY_INLINE** → harmless or accompanying actions

This visual grammar helps users recognise the seriousness of an action without reading the text. It is precisely this clarity that is indispensable when operations are applied to many data sets rather than a single one.

Conclusion: From individual case to professional mass editing

With today’s update, the URL shortener’s overview view reaches a new level of functionality. What was previously a tool for individual case management is now developing into a full-fledged interface for professional mass editing. The fundamental expansion consists not only in the introduction of several bulk operations, but above all in how these operations are embedded within a consistent operating concept.

In the previous chapters, it became apparent that the introduction of multiple selection, bulk bar, dialogue mechanisms and a finely graded visual language is not a loose juxtaposition of individual features. Instead, all elements are intertwined: the search structures the data space, the selection defines the area of action, the bulk bar makes the mass operations visible, and the dialogues ensure conscious, comprehensible decisions. Finally, visual semantics provide orientation and support safe operation even in fast or routine workflows.

This combination of architecture and interaction design provides the foundation for functions that can be added in future expansion stages, including automated workflows, grouped changes, role-based sharing, and even domain-specific mass transformations. These changes thus not only provide a feature package but also the foundation for a productive, reliable, and extensible mass-operations architecture.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-mass-grid-operations-part-1/Mon, 15 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-mass-grid-operations-part-1/The next stage of the Advent calendar focuses on further development, which becomes immediately noticeable once more shortlinks are used. The previous interaction patterns in the Overview view were geared toward individual operations and yielded only limited efficiency gains when processing many objects simultaneously. Today, this paradigm is being deliberately broken up and replaced by an interplay of new UI concepts that, for the first time, enable true multi-operation, context-sensitive work, and a much more stringent user interface.<![CDATA[

The next stage of the Advent calendar focuses on further development, which becomes immediately noticeable once more shortlinks are used. The previous interaction patterns in the Overview view were geared toward individual operations and yielded only limited efficiency gains when processing many objects simultaneously. Today, this paradigm is being deliberately broken up and replaced by an interplay of new UI concepts that, for the first time, enable true multi-operation, context-sensitive work, and a much more stringent user interface.

The source code for this article can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-08

The changes focus on three aspects: first, the introduction of a valid multiple selection, which was previously missing and now forms the basis for all new mass operations. Second, the contextual action bar, which appears only when relevant and blends seamlessly into the workflow. Third, a visibly improved depth of interaction, which is reflected in consistent visual cues, reduced refresh cycles, and the ability to perform more complex processing steps – such as setting or removing expiration dates – for any number of entries at the same time.

These improvements should not be viewed as a collection of isolated features, but rather as a coordinated step toward a more mature UI architecture. They address a common issue in production environments: users manage not just one but dozens or hundreds of shortlinks. Any optimisation that reduces friction losses here directly affects work speed and quality. This is exactly where this step has its impact: the application is not only faster to use but also more tailored to professional use cases.

Multiple selection as the foundation for mass grid operations

The transition from a single to a multiple selection marks a fundamental change in the interaction logic of the Overview view. Until now, working with shortlinks has been limited to isolated operations that always followed the same sequence: an entry was selected, edited, confirmed, and the process started over again. This structure was functional but proved to be a stumbling block when handling a larger number of entries in real-world applications.

With the introduction of multiple selection, this bottleneck will be resolved. The grid transforms from a linear list into a workspace where various objects can be brought into focus simultaneously. This capability is not only an ergonomic improvement, but also forms the technical and conceptual foundation for all subsequent mass operations. Only the possibility of marking any number of shortlinks at the same time makes context-sensitive work possible at all.

The technical anchoring of this multiple selection is initially done at a very inconspicuous point in the grid configuration. The overview view explicitly switches from single to multi-selection mode, which means that the grid is internally able to keep several entries as selected in parallel:

javaCopy

privatevoidconfigureGrid(){logger().info("configureGrid..");grid.addThemeVariants(GridVariant.LUMO_ROW_STRIPES,GridVariant.LUMO_COMPACT);grid.setHeight("70vh");grid.setSelectionMode(Grid.SelectionMode.MULTI);

With this change, the Vaadin grid’s selection mechanism will be expanded from the ground up. While in single-selection mode, the focus is technically always on exactly one entry in Grid.SelectionMode.MULTI allows multiple selections simultaneously. The basic interaction pattern mustn’t change for the user: the choice is still made via familiar gestures, such as clicks, supplemented by keyboard input as needed. The difference lies in the underlying representation: the grid now manages many marked shortlinks rather than a single active element.

To ensure that this multiple selection is not only visually effective, but also functionally reflected, it is integrated into the rest of the interface via a selection listener:

kotlinCopy

grid.addSelectionListener(event->{varall=event.getAllSelectedItems();booleanhasSelection=!all.isEmpty();bulkBar.setVisible(hasSelection);if(hasSelection){intcount=all.size();Stringlabel=count==1?"link selected":"links selected";selectionInfo.setText(count+" "+label+" on page "+currentPage);}else{selectionInfo.setText("");}bulkDeleteBtn.setEnabled(hasSelection);bulkSetExpiryBtn.setEnabled(hasSelection);bulkClearExpiryBtn.setEnabled(hasSelection);});

Here, the abstract multiple selection is converted into a concrete UI state. The getAllSelectedItems() method returns the quantity of shortlinks currently marked in the grid. From this, a boolean aggregate state hasSelection is derived, which controls whether the downstream bulk bar is visible and whether the associated action buttons are enabled. At the same time, the selection size provides immediate feedback to the user: the number and context of the selection are summarised in a text component that makes the current page and the number of marked links visible. In this way, multiple-choice is not only managed internally but also translated into a clearly traceable, state-based interaction.

Of particular relevance is the fact that multiple selection has been seamlessly integrated into the existing UI model. The choice is made via established patterns, such as checkbox clicks or keyboard shortcuts, so users can work intuitively and productively immediately without adapting. At the same time, the visual feedback remains precise: selected rows are clearly highlighted, and changes in the selection immediately update the subsequent contextual UI components.

Thus, multiple selection provides the necessary foundation for bulk operations, such as deleting, setting or removing expiration times, and other future functions, to be implemented securely and consistently. It is the architectural pivot point for transitioning a purely entry-oriented tool into a scalable management tool for professional use cases.

The bulk action bar is a context-sensitive control centre.

With the introduction of multiple selection, you can, for the first time, edit multiple short links at once. However, this capability alone is insufficient to establish a consistent, high-ergonomics mass-machining process. Only through the bulk action bar, which dynamically appears or disappears based on the current selection, does the overview view become a true hub for mass operations.

The central idea of this bar is that tools become visible only when they are semantically relevant. As long as no entry is selected, the interface remains tidy and slim. However, as soon as one or more rows have been selected, the function space opens for all operations that refer to a set of objects. This dynamic follows the principle of context-sensitive compression: the UI does not present all possible actions simultaneously, but only those that are meaningful and feasible at the current task.

In addition, the bulk bar is not only a visual addition but also has a clearly structured internal logic, which is already reflected at the field level of the view. In the overview view, the essential UI building blocks for mass operations are explicitly declared as separate components:

javaCopy

privatefinalButtonbulkDeleteBtn=newButton("Delete selected links...");privatefinalButtonbulkSetExpiryBtn=newButton("Set expiry for selected...");privatefinalButtonbulkClearExpiryBtn=newButton("Clear expiry for selected");privatefinalSpanselectionInfo=newSpan();privatefinalHorizontalLayoutbulkBar=newHorizontalLayout();

This structure makes it clear that the bulk bar is treated as an independent UI element with its own buttons and a separate information component. The actual integration takes place immediately after the search bar, so that the mass actions are visually and functionally in proximity to global navigation and filtering:

javaCopy

publicOverviewView(){add(newH2("URL Shortener – Overview"));initDataProvider();add(buildSearchBar());add(buildBulkBar());configureGrid();addListeners();addShortCuts();}

In this way, a vertical structure is created in which search, bulk bar, and grid deliberately follow one another: First, the data space is filtered; then the selected elements are contextualised via the bulk bar before the detailed interaction takes place in the grid.

The behaviour of the bulk bar itself is encapsulated in a dedicated factory method that defines both the visual appearance and the initial functional state:

javaCopy

privateComponentbuildBulkBar(){bulkDeleteBtn.addThemeVariants(LUMO_ERROR,LUMO_TERTIARY_INLINE);bulkSetExpiryBtn.addThemeVariants(LUMO_TERTIARY_INLINE);bulkClearExpiryBtn.addThemeVariants(LUMO_TERTIARY_INLINE);bulkDeleteBtn.getElement().setProperty("title","Delete all selected short links");bulkSetExpiryBtn.getElement().setProperty("title","Set the same expiry for all selected links");bulkClearExpiryBtn.getElement().setProperty("title","Remove expiry for all selected links");bulkDeleteBtn.setEnabled(false);bulkSetExpiryBtn.setEnabled(false);bulkClearExpiryBtn.setEnabled(false);bulkBar.removeAll();selectionInfo.getStyle().set("opacity","0.7");selectionInfo.getStyle().set("font-size","var(--lumo-font-size-s)");selectionInfo.getStyle().set("margin-right","var(--lumo-space-m)");bulkBar.add(selectionInfo,bulkDeleteBtn,bulkSetExpiryBtn,bulkClearExpiryBtn);bulkBar.setWidthFull();bulkBar.setDefaultVerticalComponentAlignment(Alignment.CENTER);bulkBar.setVisible(false);returnbulkBar;}

In this method, the semantic roles of the individual elements are concretised. The theme variants particularly highlight the delete button, which is provided with an error theme and thus clearly distinguishes itself from less risky operations. The tooltips also enhance the explainability of actions by precisely describing each button’s purpose. All buttons start in a deactivated state; only the selection in the grid – as shown in the previous chapter – unlocks it in a targeted manner.

The layout definition makes it clear that the bulk bar is intended to be a full-width, horizontally aligned control centre. The selectionInfo area is slightly smaller but serves as a permanent context anchor that shows the user which mass scenario they are currently in. The visibility of the entire bar is initially set tofalse and controlled solely by the grid’s selection state. This ensures the bulk bar is visible only when it is needed. It first provides precise information on the selection status by displaying the number of selected shortlinks and the current page. Subsequently, the buttons that can be executed on this basis are unlocked. This mechanism prevents operational errors and ensures the user immediately recognises which actions can currently be performed.

Bulk Delete: Secure and scalable deletion operations in the grid

With the transition to mass operations, deleting multiple shortlinks simultaneously is a key use case. The previous UI was clearly designed for deleting individual entries, which was insufficient for many professional use cases. Especially when it comes to administrative activities, migrations, or cleaning test data, there is a natural need to remove many entries in a single step. This is precisely where the new bulk delete function comes in.

The key requirement for this function is to balance two aspects: on the one hand, maximum efficiency when handling large volumes of entries, and on the other, high robustness against unintentional or incorrect entries. The system should work quickly, but never delete data carelessly. The implementation, therefore, follows a two-stage interaction pattern, deliberately designed to enable mass processing while ensuring the necessary operational safety.

The bulk delete always starts with the multiple selection in the grid. Once many shortlinks have been marked, the bulk action bar is activated, and the “Delete selected links…” option becomes available. Only when this action is deliberately chosen does a separate confirmation dialogue open. It does more than ask a simple query; it displays sample shortcodes of the selected entries, making the deletion process transparent and verifiable for the user. This preview is essential for larger datasets, as it helps validate the selection’s accuracy without reading the entire list of selected items.

The confirmation pattern is designed to trigger the deletion process in a targeted manner without interrupting the workflow. Once confirmed, all highlighted shortlinks will be deleted one by one via the client API. The architecture supports both parallel and sequential deletion patterns; the current implementation uses a sequential approach to capture errors and report them to the UI precisely. Each deletion operation is evaluated, and the user receives a summary of the results that clearly documents both successful and failed deletions.

This combination of efficiency, security and transparency makes bulk delete a robust tool in a professional context. In the following, the relevant source texts are used to concretise and explain in detail how dialogue logic, API integration and feedback mechanisms interlock to realise a precisely controlled deletion process.

The entry point for bulk deletion is the confirmBulkDeleteSelected() method in the Overview view. It integrates the grid’s current multi-selection with the dialogue logic. It prepares both the preview of the entries to be deleted and the subsequent execution of the delete operations:

javaCopy

privatevoidconfirmBulkDeleteSelected(){varselected=grid.getSelectedItems();if(selected.isEmpty()){Notification.show("No entries selected");return;}Dialogdialog=newDialog();dialog.setHeaderTitle("Delete "+selected.size()+" short links?");varexampleCodes=selected.stream().map(ShortUrlMapping::shortCode).sorted().limit(5).toList();if(!exampleCodes.isEmpty()){Stringpreview=String.join(", ",exampleCodes);if(selected.size()>5){preview+=", ...";}dialog.add(newText("Examples: "+preview));}else{dialog.add(newText("Delete selected short links?"));}Buttonconfirm=newButton("Delete",_->{intsuccess=0;intfailed=0;for(varm:selected){try{booleanok=urlShortenerClient.delete(m.shortCode());if(ok){success++;}else{failed++;}}catch(IOExceptionex){logger().error("Bulk delete failed for {}",m.shortCode(),ex);failed++;}}dialog.close();grid.deselectAll();safeRefresh();Notification.show("Deleted: "+success+" • Failed: "+failed);});confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY,LUMO_ERROR);Buttoncancel=newButton("Cancel",_->dialog.close());dialog.getFooter().add(newHorizontalLayout(confirm,cancel));dialog.open();}

The first block already clarifies the implementation’s security concept. If, contrary to expectations, no selection is made, the process will be aborted with an explicit notification. The actual dialogue instance receives a header that explicitly shows the number of affected shortlinks, thereby clarifying the scope of the action.

Particularly interesting is how the preview of the entries to be deleted is generated. From the number of selected shortlinks, a sorted list of your shortcodes is created, which is limited to a maximum of five examples. This limitation ensures that the dialogue remains readable even with huge selections, while an appended ellipsis makes it clear that other elements are affected. This creates a concise but meaningful presentation of the selection.

Finally, the innerConfirm handler performs the actual deletion process. For each shortlink selected, the client API of the URL shortener backend is called viaurlShortenerClient.delete(m.shortCode()). Successful and failed operations are counted; Exceptions are explicitly logged and taken into account as errors in the count. This count serves as the basis for the final user notification, which summarises the number of successfully deleted and failed entries.

After the loop completes, the dialogue is closed, the grid selection is cancelled, and a consistent update of the displayed data is triggered viasafeRefresh(). The final notification provides precise, aggregated feedback on the operation outcome, thereby completing the interaction cycle of selection, confirmation, and presentation of results.

In addition, the bulk delete function can also be operated via the keyboard. The user can use a global shortcut to trigger the delete dialogue directly, provided that a selection has been made:

javaCopy

privatevoidaddShortCuts(){UIcurrent=UI.getCurrent();current.addShortcutListener(_->{if(globalSearch.isEnabled())globalSearch.focus();},Key.KEY_K,KeyModifier.META);current.addShortcutListener(_->{if(!grid.getSelectedItems().isEmpty()){confirmBulkDeleteSelected();}},Key.DELETE);}

This integrates the bulk delete into the everyday workflow: pressing the Delete key triggers the same confirmation dialogue as clicking the corresponding button in the bulk bar. Mouse and keyboard interaction thus yields the same result, significantly speeding up operations for experienced users without compromising the security of the operation.

Bulk Set Expiry: Uniform expiration dates for multiple shortlinks

With the possibility of selecting several entries at the same time, it is possible for the first time to make even more complex metadata changes in a single step. One of the most functionally essential operations in this context is setting uniform expiration dates for multiple shortlinks. This feature is particularly relevant in administrative scenarios, such as when a large number of temporary links are set to expire simultaneously or when test or campaign data needs to be centrally controlled.

The design of the bulk set expiry function is intended to be both precise and efficient. Users should be able to set the date and time in a consistent dialogue without having to navigate through each entry individually. At the same time, the implementation must robustly handle erroneous inputs and provide clear feedback on the operation’s success or failure.

The dialogue-based setting of a common expiration date follows a three-step interaction pattern. First, the user selects all relevant shortlinks, which makes the bulk bar visible and enables the “Set expiry for selected…” option. In the second step, a custom dialogue opens to enter the date and time. Only in the third step, after the conscious confirmation, are the new expiration dates set for all selected entries and the grid is updated accordingly.

The entry point into this functionality is theopenBulkSetExpiryDialog() method, which is triggered directly from the bulk action bar. It encapsulates both the dialogue-based recording of the expiration time and the later forwarding to the server API:

javaCopy

privatevoidopenBulkSetExpiryDialog(){varselected=grid.getSelectedItems();if(selected.isEmpty()){Notification.show("No entries selected");return;}Dialogdialog=newDialog();dialog.setHeaderTitle("Set expiry for "+selected.size()+" short links");DatePickerdate=newDatePicker("Date");TimePickertime=newTimePicker("Time");time.setStep(Duration.ofMinutes(15));HorizontalLayoutbody=newHorizontalLayout(date,time);body.setAlignItems(Alignment.END);dialog.add(body);Buttoncancel=newButton("Cancel",_->dialog.close());Buttonapply=newButton("Apply",_->{if(date.getValue()==null){Notification.show("Please select a date");return;}varlocalTime=Optional.ofNullable(time.getValue()).orElse(LocalTime.of(0,0));varzdt=ZonedDateTime.of(date.getValue(),localTime,ZoneId.systemDefault());InstantexpiresAt=zdt.toInstant();intsuccess=0;intfailed=0;for(ShortUrlMappingm:selected){try{booleanok=urlShortenerClient.edit(m.shortCode(),m.originalUrl(),expiresAt);if(ok)success++;elsefailed++;}catch(IOExceptionex){logger().error("Bulk set expiry failed for {}",m.shortCode(),ex);failed++;}}dialog.close();grid.deselectAll();safeRefresh();Notification.show("Updated expiry: "+success+" • Failed: "+failed);});apply.addThemeVariants(ButtonVariant.LUMO_PRIMARY);dialog.getFooter().add(newHorizontalLayout(cancel,apply));dialog.open();}

The method header already includes a security prompt that prevents the dialogue from opening accidentally without a valid selection. The dialogue configuration is deliberately kept minimalist and focuses on the core input elements: aDatePicker for the date and aTimePicker for the time. The TimePicker’s step size is set to 15 minutes, which is a practical compromise between precision and usability.

The core logic begins with validating the date. If no date is selected, the execution is cancelled, and a corresponding notification is issued. If a date is specified, the method creates aZonedDateTime from it, which is then converted to aninstant. Thisinstant represents the new expiration time to be assigned to all selected shortlinks.

The following loop block illustrates the interface to the backend. For each shortlink, the edit operation is called:

javaCopy

urlShortenerClient.edit(m.shortCode(),m.originalUrl(),expiresAt);

This operation is designed to change only the expiration time, while preserving the original URL value. Errors are handled individually and accounted for in both the UI and logging. Summing up the successful and failed updates results in aggregated feedback at the end of the process, which immediately tells the user how successful the bulk operation was.

Finally,grid.deselectAll() combined with safeRefresh() ensures that both the UI state and the grid data are updated consistently. The user is thus placed in a clean reset state without any manual intermediate steps, while the final Notification.show(…) summarises the operation results. The bulk set expiry function illustrates the interplay among the UI, server API, and error handling, and explains how this mechanism greatly simplifies managing large numbers of links.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-from-ui-interactions-to-a-deterministic-refresh-architecture/Sun, 14 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-from-ui-interactions-to-a-deterministic-refresh-architecture/After the first part explained the conceptual basics and the new interactions among global search, search scopes, and advanced filters, the second part focuses on the technical mechanisms that enable these interactions. It is only the revised refresh architecture – above all the interaction of safeRefresh() and RefreshGuard – that ensures that the OverviewView remains calm, deterministic and predictable despite numerous potential triggers.<![CDATA[

After the first part explained the conceptual basics and the new interactions among global search, search scopes, and advanced filters, the second part focuses on the technical mechanisms that enable these interactions. It is only the revised refresh architecture – above all the interaction ofsafeRefresh() andRefreshGuard – that ensures that the OverviewView remains calm, deterministic and predictable despite numerous potential triggers.

So while Part 1 describeswhat has changed in the user interface andwhy this structuring was necessary, Part 2 now shows in detailhow the internal state machine works, how competing UI events are coordinated and why the View achieves the desired robustness in the first place.

With this foundation, the following chapters can be assigned clearly: they analyse the refresh architecture, the reset mechanism, and the validation and error-handling logic – the technical building blocks that ensure the UI concepts described above not only look good but also function reliably.

The source code for this development step can be found on GitHub

and can be found here:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-07

Here are some screenshots of the current development state.

Refresh architecture with RefreshGuard

With the introduction of advanced filtering logic, not only does the complexity of the user interface increase, but so does the number of potential conditions that can cause a grid reload. Any change to a filter field, any adjustment to the page size, the opening or closing of the Advanced area, or entering the view itself can trigger a new request to the server. Without additional control, this could trigger a cascade of overlapping refreshes, create unnecessary load, and noticeably degrade the user experience.

Against this background, the refresh architecture of the OverviewView has been specifically revised. The central goal was to reduce the number of possible triggers to a controllable mechanism that exhibits consistent, predictable behaviour across all relevant scenarios. Instead of allowing each UI event to have its own access to the DataProvider, a deliberate hard cut was made: the decision of whether and when a refresh may actually take place is bundled in a dedicated layer. This layer contains theRefreshGuard, which acts as a guardian that determines which processes are allowed to retrieve specific data and which are only allowed to change the internal state.

The technical basis of this control system is surprisingly simple at first. A single flag is introduced at the head of the class that controls whether refreshes are currently allowed:

javaCopy

privatebooleansuppressRefresh=false;

This flag is evaluated by a small helper method that acts as the only allowed entry point for data updates:

javaCopy

privatevoidsafeRefresh(){logger().info("safeRefresh");if(!suppressRefresh){logger().info("refresh");dataProvider.refreshAll();}}

Instead of reloading the grid or DataProvider from multiple locations, the view periodically callssafeRefresh(). The method first logs the refresh attempt, then checks the flag. Only ifsuppressRefresh is not set,refreshAll() is actually executed on the DataProvider. This creates a clear separation between the semantic intent to update the content and the operational decision of whether it is allowed in the current context.

Conceptually, theRefreshGuard can be understood as a critical section that allows multiple UI operations to be combined. As long as the code is within such a protected phase, direct refresh calls are suppressed. Only at the end of the section does the guard decide whether an aggregated refresh is required and, if so, execute it in a controlled manner. In the source code, this mechanism is implemented as a small, inner helper class:

javaCopy

classRefreshGuardimplementsAutoCloseable{privatefinalbooleanprev;privatefinalbooleanrefreshAfter;RefreshGuard(booleanrefreshAfter){this.prev=suppressRefresh;this.refreshAfter=refreshAfter;suppressRefresh=true;}@Overridepublicvoidclose(){suppressRefresh=prev;if(refreshAfter)safeRefresh();}}

The constructor of the guard first stores the current state ofsuppressRefresh and then sets the flag totrue. This means that allsafeRefresh() operations called within this block are ineffective; they are logged but do not trigger a reload. When leaving the block (i.e., in the close() method), the previous flag value is restored. Optionally, a final refresh can then be triggered by therefreshAfter parameter. By implementingAutoCloseable, the guard can be used in a try-with-resources block, ensuring that the state is reset correctly even in exceptional cases.

The use of this pattern is particularly evident in the View lifecycle. When attaching the OverviewView to the UI, the column visibilities are first set based on the stored preferences. This process changes the grid state significantly but should not cause new data to be loaded from the server multiple times. Accordingly, the entire block is wrapped in aRefreshGuard :

javaCopy

@OverrideprotectedvoidonAttach(AttachEventattachEvent){super.onAttach(attachEvent);try(var_=newRefreshGuard(true)){varkeys=grid.getColumns().stream().map(Grid.Column::getKey).filter(Objects::nonNull).toList();varvis=columnVisibilityService.mergeWithDefaults(keys);grid.getColumns().forEach(c->{vark=c.getKey();if(k!=null)c.setVisible(vis.getOrDefault(k,true));});}subscription=StoreEvents.subscribe(_->getUI().ifPresent(ui->ui.access(this::safeRefresh)));}

Within the try block,suppressRefresh is set totrue; all grid operations run without immediate data retrieval. Only when leaving the block and all column visibilities are set consistently does the guard provide a single final refresh withrefreshAfter = true. As a result, the view is loaded exactly once after the first setup, rather than flickering through intermediate states during initialisation.

safeRefresh() also demonstrates its strengths when navigating between pages. The buttons for scrolling forward and backwards only change the internal page cursor and explicitly delegate the data update to the central method:

javaCopy

prevBtn.addClickListener(_->{if(currentPage>1){currentPage--;refreshPageInfo();safeRefresh();}});nextBtn.addClickListener(_->{intsize=Optional.ofNullable(pageSize.getValue()).orElse(25);intmaxPage=Math.max(1,(int)Math.ceil((double)totalCount/size));if(currentPage<maxPage){currentPage++;refreshPageInfo();safeRefresh();}});pageSize.addValueChangeListener(e->{currentPage=1;grid.setPageSize(Optional.ofNullable(e.getValue()).orElse(25));safeRefresh();});

Here, the use of the guard is deliberately avoided, as each of these actions constitutes a clearly demarcated, independent refresh. Scrolling back or forward to a new page section, or flipping forward, should trigger a change in page size immediately. Nevertheless, all accesses continue to run viasafeRefresh(), so that the mechanism remains centrally controllable. If the architecture changes in the future, an adjustment at this point is sufficient to modify the behaviour of the entire view consistently.

Taken together, the combination ofsafeRefresh() andRefreshGuard transforms the refresh behaviour of the OverviewView from a hard-to-predict byproduct of many UI events to a controlled, deterministic strategy. Complex operations such as initialisation and reset are packed into closed, atomic blocks, while simple actions such as page changes and field changes are explicitly allowed to trigger a refresh. This gives the view both stability and transparency: readers of the code can clearly see where data updates occur, and users of the interface perceive a quiet, responsive application that responds to inputs in a comprehensible way.

Reset Mechanism: Full State-Clear

The reset mechanism of the OverviewView plays a special role within the search and filter architecture. It forms the fastest way back to a clearly defined, neutral initial state – a state in which neither search fragments nor extended filters, sorting options, nor deviating page sizes are active. While earlier implementations often reset only partial aspects, the revised version takes a consistently holistic approach: clicking “Reset” deletes all user-changed parameters without exception and resets the view as if it were being opened for the first time.

Conceptually, this mechanism is directly integrated into the previously introduced refresh architecture. Since the reset involves a large number of individual steps – emptying text fields, resetting checkboxes and date entries, restoring the default sorting, resetting the page size and closing the Advanced area – each of these actions would immediately trigger a refresh when viewed in isolation. To prevent this, and to treat all changes as logical operations, the entire reset process is embedded within a RefreshGuard.

The reset button implementation directly reflects these considerations. The listener for the reset button encapsulates all the necessary steps in a single, clearly defined block:

javaCopy

resetBtn.addClickListener(_->{try(var_=newRefreshGuard(true)){globalSearch.clear();codePart.clear();codeCase.clear();urlPart.clear();urlCase.clear();fromDate.clear();fromTime.clear();toDate.clear();toTime.clear();sortBy.clear();dir.clear();pageSize.setValue(25);sortBy.setValue("createdAt");dir.setValue("desc");currentPage=1;searchScope.setValue("URL");advanced.setOpened(false);setSimpleSearchEnabled(true);globalSearch.focus();}});

The reset process begins by clamping a RefreshGuard with the parameter set totrue. This first sets thesuppressRefresh flag, so that allsafeRefresh() callsindirectly triggered in this block‑remain ineffective. Only when leaving the block (controlled by refreshAfter = true) is a final refresh executed, making the cumulative new state visible in the grid.

Within the block, all user inputs are systematically returned to their original state. First, all search and filter fields are cleared: the global search field, the shortcode and URL fragments in the Advanced area, and the case-sensitivity checkboxes. The date and time fields for the time window under consideration are then set tozero. This ensures that no old period inadvertently affects later requests.

In the next step, the sorting is reset to its default values. First,sortBy anddir are removed to avoid potential inconsistencies; then,createdAt is explicitly setas the sort field anddesc as the sort direction. The page size is also deliberately set back to the default value of 25 entries per page, and the page cursorcurrentPage is set to one. This creates a state that corresponds to the first time you enter the view: no running filters, a defined sorting and a comprehensible page setting.

The global search logic is also reinitialised. The search scope is reset to URL, and the Advanced scope is closed byAdvanced.setOpened(false ). CallingsetSimpleSearchEnabled(true) re-enables simple mode, andglobalSearch.focus() ensures that the cursor lands directly in the global search field. This results in an intuitive process for the user: After the reset, he sees a neutral overview and can immediately start a new, simple search.

This keeps the user interface completely quiet during the reset: no flickering, no multiple queries, no inconsistent intermediate layout. Only when the entire process is complete is a single, final refresh executed, which establishes a consistent initial state across the grid. This stability is not only crucial for the user experience but also facilitates extensibility, as additional reset steps can be added without introducing new side effects. As long as new fields are included in this guard block, they remain part of the same atomic operation.

In combination with the search and filter logic, this results in a reset mechanism that is both semantically and technically cleanly modelled. Semantic because the user has a clear expectation – “everything back to square one” – that is fully met. Technically, because the mechanism is embedded in the central refresh architecture by the RefreshGuard, it causes neither uncontrolled side effects nor hidden data retrievals. On this basis, subsequent chapters can now address error cases, validations, and logging in a more granular way without touching the basic reset path again.

Error handling, validation, and robustness

With the growing number of functions in OverviewView, robustness is increasingly critical. The user interface should not only work reliably in ideal scenarios but also handle incomplete, contradictory, or incorrect inputs. The combination of global search, extended filter area, time windows, sorting settings and page size in particular presents the system with the challenge of recognising and stably handling even complex and potentially conflict-prone states.

In the revised architecture, robustness is not treated as an afterthought, but as an integral part of the UI logic. Many validation and error-avoidance mechanisms are deeply embedded in interaction points: in ValueChange listeners, when switching between Simple and Advanced modes, and when resetting and deriving a binding search string. The view does not aim to take away all user freedom, but instead offers a controlled environment in which only consistent states can arise. The technical side deliberately avoids complex error messages in favour of clear, deterministic rules that become directly visible in the interaction of the input elements.

A central element of this robust logic is the automatic verification of consistency for the search and filter fields. The pattern of defensive synchronisation is already clearly evident in the global search field:

kotlinCopy

globalSearch.addValueChangeListener(e->{varv=Optional.ofNullable(e.getValue()).orElse("");if(searchScope.getValue().equals("Shortcode")){codePart.setValue(v);urlPart.clear();}else{urlPart.setValue(v);codePart.clear();}});

This logic ensures that the shortcode and URL fragment cannot be active simultaneously. As soon as the user enters something in the global search box, the value is interpreted as either a shortcode or a URL. The other field is consistently emptied. In this way, there are no conflicting filter states that would force the application to fulfil two search intentions simultaneously.

The Scope Selection listener reinforces this rule by ensuring that even subsequent changes to the search scope always result in a consistent state:

kotlinCopy

searchScope.addValueChangeListener(_->{varv=Optional.ofNullable(globalSearch.getValue()).orElse("");if("Shortcode".equals(searchScope.getValue())){codePart.setValue(v);urlPart.clear();}else{urlPart.setValue(v);codePart.clear();}});

This prevents a user from typing a search query in URL mode, then switching to shortcode, thereby implicitly creating an invalid search model. The UI detects this state early and maps it to a clear, comprehensible model.

The validation and cleanup mechanisms are particularly evident in advanced mode when deriving a valid simple search state. TheapplyAdvancedToSimpleAndReset() method is the technical condensation of this approach:

javaCopy

privatevoidapplyAdvancedToSimpleAndReset(){Stringcode=Optional.ofNullable(codePart.getValue()).orElse("").trim();Stringurl=Optional.ofNullable(urlPart.getValue()).orElse("").trim();finalbooleanhasCode=!code.isBlank();finalbooleanhasUrl=!url.isBlank();finalStringwinnerValue=hasCode?code:(hasUrl?url:"");finalStringwinnerScope=hasCode?"Shortcode":"URL";try(var_=newRefreshGuard(true)){codePart.clear();codeCase.clear();urlPart.clear();urlCase.clear();fromDate.clear();fromTime.clear();toDate.clear();toTime.clear();sortBy.clear();dir.clear();sortBy.setValue("createdAt");dir.setValue("desc");searchScope.setValue(winnerScope);if(!winnerValue.isBlank()){globalSearch.setValue(winnerValue);}else{globalSearch.clear();}setSimpleSearchEnabled(true);globalSearch.focus();}}

Several basic principles of robustness are intertwined here. First, it checks whether a shortcode or a URL fragment is set. If both are present, the shortcode fragment is given priority – a clear and comprehensible rule that avoids ambiguity. The entire Advanced area is then consistently cleaned up to ensure that no old or partially set values are unintentionally included in future filters.

In addition,RefreshGuard plays a special role in robust processes. Without the guard, the numerous changes within this method would trigger a series of refresh events. However, the guard suppresses these events selectively and triggers exactly one consistent refresh at the end. This prevents flickering UI transitions and ensures the user always sees the final state.

Another important component is validating time windows. The combination of DatePicker and TimePicker can naturally generate incomplete entries – such as a set date without time or vice versa. The logic in the backend transport takes care of these cases, but already in the UI code, a defensive determination of the timestamp prevents potential errors:

xmlCopy

private Optional<Instant> combineDateTime(DatePicker date, TimePicker time) { var d = date.getValue(); var t = time.getValue(); if (d == null&& t == null) return Optional.empty(); if (d == null) return Optional.empty(); var lt = (t != null) ? t : LocalTime.MIDNIGHT; return Optional.of(lt.atDate(d).atZone(ZoneId.systemDefault()).toInstant());}

The method is generous, but clear: a time value without a date is not a valid filter. In case of doubt, times are set to midnight, which keeps the model stable even in incomplete scenarios. This type of defensive modelling prevents incomplete UI inputs from leading to inconsistent backend requests.

The event handlers provide additional technical protection for paging and navigation. Actions such as scrolling between pages or changing the page size have a direct effect on the database, but should not trigger any unexpected side effects. The consistent use ofsafeRefresh() ensures that these changes only take effect if the refresh context allows it:

javaCopy

pageSize.addValueChangeListener(e->{currentPage=1;grid.setPageSize(Optional.ofNullable(e.getValue()).orElse(25));safeRefresh();});

Here, too, robustness is created by clear rules: A new page size resets the page cursor and triggers a controlled reload – never several, never via a detour.

Finally, logging also contributes significantly to diagnostic robustness. In many places in the code,logger().info() is deliberately used to signal when search changes, refresh processes, or state transitions occur. These traces enable precise reconstruction of complex error patterns in UI-backend interactions without requiring additional debugging mechanisms.

The result is a system that does not rely on errors that are subsequently intercepted, but on deliberately modelled, conflict-free states. The user guidance is designed to prevent invalid or contradictory situations, and the technical foundation ensures that even incomplete or ambiguous inputs are converted into stable, controlled processes. Thus, the combination of validation, synchronisation, and protection mechanisms provides a viable basis for further expansion of the application.

Conclusion and outlook

The revision to the OverviewView is more than a collection of individual improvements. It marks a structural change that fundamentally reshapes the interaction among search, filtering, data updates, and user interaction. From an initially heterogeneous interface with scattered responsibilities, a clearly modelled, consistent and technically robust view has emerged, whose behaviour is comprehensible and extensible in all essential dimensions.

A key result of this revision is the standardisation of the search logic. The introduction of a global search box, together with scope selection, forms a small, self-contained state machine that provides an intuitive entry point for the user. Complemented by the extended filter area, a flexible system is created that supports both fast search queries and more complex filter combinations – without competing with each other. The clear switch between advanced and straightforward mode prevents contradictory states and keeps cognitive effort low.

Equally important is the redesigned refresh architecture. WithsafeRefresh() andthe RefreshGuard, a mechanism has been introduced to stabilise the application’s refresh behaviour. Complex operations such as initialisation or reset are bundled into atomic, deterministic processes, while simple interactions can still react directly. This pattern operates in the background and is particularly noticeable to the user when the operation is quieter and less erratic.

The grid itself has also been further developed in terms of functionality and ergonomics. Copy functions, context-sensitive opening of details, dynamic flow badges and an improved column layout transform the table into an interactive workspace that not only provides information, but also enables action. This proximity between data and interaction reduces the need for additional dialogue changes, thereby contributing to a more fluid workflow.

The robustness of the view ultimately results from a variety of small but effective mechanisms: automatic synchronisation of filter fields, defensive evaluation of incomplete entries, clear prioritisation rules, and consistent logging. All these aspects ensure that the application remains reliable even under unusual input combinations and that the causes of errors can be traced if necessary.

This structural basis enables broad future expansion. The clear separation of UI logic, filter model, and refresh strategy provides a stable foundation on which subsequent features can be built without breaking points. Among other things, the following are conceivable:

– a server-side full-text search that extends the Simple/Advanced model, – colour or iconographic markings of other states such as “soon to expire”, – bulk actions for multiple selections, – a modular sorting and filtering pipeline, – tagging or labelling functions for short URLs, – advanced column settings or custom views.

The new OverviewView thus not only represents an improvement over the status quo but also marks a structural turning point. It creates clarity where implicit or scattered logic previously prevailed and establishes mechanisms that keep the system stable over the long term. In its entirety, the revision represents an important step towards a modern, scalable, and maintainable UI architecture that meets the requirements of growing use cases.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-from-simple-search-to-expert-mode-advanced-filters-and-synchronised-scopes-for-power-users/Sat, 13 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-from-simple-search-to-expert-mode-advanced-filters-and-synchronised-scopes-for-power-users/Since its inception, the URL shortener’s continuous development has focused on two core goals: a robust technical foundation without external frameworks and a modern, productive user interface that is both intuitive and efficient for power users. As part of the current development stage, an essential UI module has been revised – the OverviewView, i.e. the view in which users search, filter and manage all saved shortenings.<![CDATA[

Since its inception, the URL shortener’s continuous development has focused on two core goals: a robust technical foundation without external frameworks and a modern, productive user interface that is both intuitive and efficient for power users. As part of the current development stage, an essential UI module has been revised – the OverviewView, i.e. the view in which users search, filter and manage all saved shortenings.

In the previous version, it became increasingly clear that the search and filtering components were decoupled. The basic search offered limited functionality, while the advanced filters were present but not integrated into a real control flow. In addition, the interaction between user actions such as reset, filter changes, paging, or opening the detail view was not sufficiently stabilised, which sometimes led to multiple refreshes or contradictory UI states.

The revision now implemented aims to achieve several structural objectives. The central focus was on standardising interactions: the search function was redesigned and integrated into a unified concept, including a global search bar with a clearly defined scope and a structured area for advanced filters. In parallel, the technical architecture of the refresh mechanisms has been revised to deliver a quieter, more reliable user experience.

This introduction first outlines the rationale for the changes and situates them within the broader development context. In the following chapters, the new global search is first described, followed by the synchronisation mechanisms, extended filters, internal refresh architecture, and grid improvements. The goal is not only a functional description but also a technical analysis of the mechanisms that enable these improvements and underpin future expansions.

The source code for this development step can be found on GitHub

and can be found here:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-07

The new global search

The introduction of global search marks a central step towards a consistent yet flexible operating logic within the OverviewView. While in the previous version the search function consisted of several independent elements, a uniform interaction surface has now been created, whose behaviour is clearly defined and technically cleanly modelled. The basis is a single text field, supplemented by a search-area selection, which eliminates the prior separation between URL and shortcode searches.

In the source code, this standardisation is first reflected in the explicit introduction of two central UI components, which are declared as fixed components of the view:

xmlCopy

private final TextField globalSearch = new TextField();private final ComboBox<String> searchScope = new ComboBox<>("Search in");private final Button advancedBtn = new Button("Advanced filters", new Icon(VaadinIcon.SLIDERS));

This clearly defines that there is precisely one global search field and exactly one selection for the search area. Both components are created early in the view’s life cycle and are therefore available to all subsequent configuration steps. The actual design is done in thebuildSearchBar() method, in which placeholders, width, and interaction behavior are specifically defined:

javaCopy

privateComponentbuildSearchBar(){globalSearch.setPlaceholder("Search all...");globalSearch.setClearButtonVisible(true);globalSearch.setWidth("28rem");globalSearch.setValueChangeMode(LAZY);globalSearch.setValueChangeTimeout(VALUE_CHANGE_TIMEOUT);searchScope.setItems("URL","Shortcode");searchScope.setValue("URL");searchScope.setWidth("11rem");pageSize.setMin(1);pageSize.setMax(500);pageSize.setStepButtonsVisible(true);pageSize.setWidth("140px");// ...}

The global search is not treated as an arbitrary text field, but is deliberately modelled as a central control element. The placeholder “Search all…” makes it clear that, regardless of the specific search area, the user initially enters only one search term. The actual routing of this value into the appropriate technical field handles the logic associated with the search field and scope selection. Choosing LAZY mode, combined with an explicit timeout, ensures that server-side filter requests are triggered only when the user has completed input.

The search box serves as the starting point for all queries. As soon as the user enters, the content is assigned to either the URL or the shortcode component of the filter model, depending on the currently selected search area. This assignment is not only a UI-side mechanism but also a clear rule within the internal logic: the value of the global search field is always bound to exactly one of the two fields in the request object. The central implementation of this coupling is carried out via theValueChangeListener of the global search field:

kotlinCopy

globalSearch.addValueChangeListener(e->{varv=Optional.ofNullable(e.getValue()).orElse("");if(searchScope.getValue().equals("Shortcode")){codePart.setValue(v);urlPart.clear();}else{urlPart.setValue(v);codePart.clear();}});

Here, each new value is first converted to a safe, non-null variant. The value of the ComboBoxsearchScopethen decideswhether the search term is interpreted as a shortcode filter (codePart) or a URL filter (urlPart). Only one of the two fields may be occupied at a time; the other is cleared automatically. This avoids ambiguous search situations in which multiple filters are applied simultaneously and uncoordinatedly. The global search field thus becomes the sole source for exactly one specific logical filter state.

Another major innovation is the tight coupling between the search field and the search area. The selection of the range – URL or shortcode – directly determines which part of the filter model is active. To ensure that this relationship remains consistent in both directions, the scope selection also reacts to changes and reflects the current search value in the appropriate field:

kotlinCopy

searchScope.addValueChangeListener(_->{varv=Optional.ofNullable(globalSearch.getValue()).orElse("");if("Shortcode".equals(searchScope.getValue())){codePart.setValue(v);urlPart.clear();}else{urlPart.setValue(v);codePart.clear();}});

While the text field listener reacts when the search term changes, it is responsible for maintaining consistent search state when the user subsequently changes the search scope. Both implementations follow the same pattern: a standard source value is interpreted as either a shortcode or a URL, and the inactive field is consistently emptied. This keeps the search interface not only visually clear, but also logical.

The behaviour of the global search is also designed to enable clear prioritisation when combined with the advanced filters. As long as the advanced filters are closed, the global search box controls the filter state independently. When the user opens the advanced area, the global search loses its active role and is relegated to the background, both visually and technically. This separation is encapsulated by a small helper method that explicitly determines the state in which the simple search may be:

javaCopy

privatevoidsetSimpleSearchEnabled(booleanenabled){globalSearch.setEnabled(enabled);searchScope.setEnabled(enabled);resetBtn.setEnabled(true);globalSearch.setHelperText(enabled?null:"Disabled while Advanced filters are open");}

This method not only controls the activation and deactivation of the input elements but also provides context-sensitive help text that explains why global search is unavailable in open Advanced mode. This prevents two parallel filter sources from competing with each other and destabilizing the overall state. At the same time, the operating concept remains transparent, as the UI clearly communicates its status.

With this revamped global search, an intuitive, clear entry point has been created with a well-defined function from both the user and technical architecture perspectives. The search field, the scope selection, and the associated filter fields form a small, self-contained state machine whose behaviour is explicitly modelled in the code. The following chapters now examine how this search interacts with the other components and how the underlying synchronisation logic ensures consistent state transitions.

Search scopes and synchronisation logic

While the global search offers a clear entry point, its real strength only becomes apparent through interaction with the underlying synchronisation logic. It is crucial that the selected search scope – i.e. the decision between URL and shortcode – does not remain just a visual interface detail, but is consistently transferred to the internal state model. The goal of this layer is to ensure that at any given time, it is clear which filter is active and which parts of the UI represent that filter.

From the user’s perspective, the behaviour can be divided into two central scenarios. In simple mode, the combination of the global search field and scope selection directly controls the filter state. The user implicitly determines, via the input context, whether to search for a target URL or a shortcode. In advanced mode, by contrast, the global search is reversed, and the detail fields fully control the filter state. This transition between modes is the core of synchronisation logic.

From a technical standpoint, this logic is based on a few, clearly defined principles. First, there is exactly one source for the effective search string at any given time. In simple mode, this is the global search field, which is mapped to either the URL or the shortcode component of the filter model, depending on the scope. In advanced mode, the dedicated URL and shortcode fields in the Advanced area are transferred directly to the request object. Second, there must be no competing states: if the user is working in Advanced mode, global search items are disabled; when the user returns to simple mode, the state is derived from the previous detail values.

The technical implementation of this switch begins where the View establishes the Advanced area as a controlling element. Central is the listener, which reacts to the opening and closing of thedetails container:

javaCopy

advanced.addOpenedChangeListener(ev->{booleannowClosed=!ev.isOpened();if(nowClosed){applyAdvancedToSimpleAndReset();}else{setSimpleSearchEnabled(false);}});

These few lines model the entire state change between the modes. When the user opens the Advanced section, the simple search is disabled. When closing, not only is the Advanced area collapsed, but a consolidation step is also triggered viaapplyAdvancedToSimpleAndReset(), which converts the previous detailed configuration into a simple, global search state.

To ensure that disabling the simple search does not lead to an inconsistent UI impression, the View encapsulates the necessary adjustments in a small helper method:

javaCopy

privatevoidsetSimpleSearchEnabled(booleanenabled){globalSearch.setEnabled(enabled);searchScope.setEnabled(enabled);resetBtn.setEnabled(true);globalSearch.setHelperText(enabled?null:"Disabled while Advanced filters are open");}

The method controls both the interactivity of the search field and scope selection as well as the accompanying help text. Once the Advanced section is opened, the Basic Search values are retained, but cannot be changed. At the same time, the helper text makes it clear that global search is currently disabled. At this level, the first part of the principle described above is implemented: There is always only one active source that determines the effective filter state.

The opposite direction – from advanced mode back to simple mode – is more complex because a choice has to be made here. In the Advanced area, a shortcode fragment and a URL fragment can be entered at the same time. Both would be suitable as filters but cannot be readily combined into a single global search field. This is precisely whereapplyAdvancedToSimpleAndReset() comes in:

javaCopy

privatevoidapplyAdvancedToSimpleAndReset(){Stringcode=Optional.ofNullable(codePart.getValue()).orElse("").trim();Stringurl=Optional.ofNullable(urlPart.getValue()).orElse("").trim();finalbooleanhasCode=!code.isBlank();finalbooleanhasUrl=!url.isBlank();finalStringwinnerValue=hasCode?code:(hasUrl?url:"");finalStringwinnerScope=hasCode?"Shortcode":"URL";try(var_=newRefreshGuard(true)){codePart.clear();codeCase.clear();urlPart.clear();urlCase.clear();fromDate.clear();fromTime.clear();toDate.clear();toTime.clear();sortBy.clear();dir.clear();sortBy.setValue("createdAt");dir.setValue("desc");searchScope.setValue(winnerScope);if(!winnerValue.isBlank()){globalSearch.setValue(winnerValue);}else{globalSearch.clear();}setSimpleSearchEnabled(true);globalSearch.focus();}}

The method begins by evaluating the codePart andurlPart detail fields. Both values are defensively converted to strings and then checked for non-empty content. Two things are derived from this: a “winner” value and a “winner” scope. If a shortcode fragment is set, it takes precedence over any URL fragment. Only if there is no shortcode and only a URL is the URL considered a winner. If both are empty, an empty search string is used, and the scope is reset to “URL”. In this way, the prioritisation described in the running text is implemented in practice, without ambiguity.

In the second block of the method, all Advanced fields are consistently reset. In addition to the text fields for shortcodes and URLs, this applies to the case-sensitivity checkboxes and the time-slot and sorting fields. The Advanced range is thus returned to a defined initial state. Thanks to theRefreshGuard, this reset does not occur through multiple individual refreshes; instead, it is treated as an aggregated state change that culminates in a controlled reload.

Only then is the previously determined winner state reflected into the simple search. The global search scope is set to winnerScope; the global search string is either filled with winnerValue or left empty. Finally, the simple search is reactivated, and the focus is set to the global search field. This provides the user with a straightforward, reduced interface after closing the Advanced area, reflecting the active filter state derived from the previously selected detail values.

Overall, this yields a small but precise state machine. Opening the Advanced pane shifts control entirely to the detail fields and visibly disables global search. Closing triggers a controlled reduction to a single, easy-to-understand filter state. The synchronisation logic remains fully comprehensible in the code, is encapsulated in a few clearly structured methods, and can be extended with additional filter fields if necessary without violating the basic principle. On this basis, the following chapters can now examine other aspects of filtering in detail, such as the extended filter fields and the refresh architecture.

Advanced Filters: Conception and UI Design

The global search serves as a compact entry point to the OverviewView’s filter logic. However, their range of functions is deliberately limited to ensure a low barrier to entry and rapid operation. As soon as the requirements go beyond a simple text fragment, this model reaches its natural limits. This is where Advanced Filters come into play, giving users much finer control over short URL filtering while providing a structured, visually comprehensible interface.

Conceptually, the Advanced area was designed as a deliberately separate mode. It should not be understood as a mere extension of the existing search field, but rather as an independent filter context that becomes active only when the user explicitly opens it. This decision takes into account two considerations. On the one hand, the simple mode should not be burdened with options unnecessary in many everyday scenarios. On the other hand, advanced filter operations – such as the combination of a shortcode fragment, a URL substring, a time window, and sorting – should have a clearly identifiable workspace in which all associated input elements are spatially bundled.

In the source code, this concept already begins at the field level, where the components for the Advanced area are clearly declared separate from the global search:

xmlCopy

private final TextField codePart = new TextField("Shortcode contains");private final Checkbox codeCase = new Checkbox("Case-sensitive");private final TextField urlPart = new TextField("Original URL contains");private final Checkbox urlCase = new Checkbox("Case-sensitive");private final DatePicker fromDate = new DatePicker("From (local)");private final TimePicker fromTime = new TimePicker("Time");private final DatePicker toDate = new DatePicker("To (local)");private final TimePicker toTime = new TimePicker("Time");private final ComboBox<String> sortBy = new ComboBox<>("Sort by");private final ComboBox<String>dir = new ComboBox<>("Direction");

These fields define the semantic dimensions of the advanced filters: textual filtering via shortcode and original URL (optional), an explicit time window, and sorting criteria. The fact that they are listed as separate attributes of the view expresses the separate mode described above: they do not belong to the global search but to an individual, extended view of the data space.

From a UI perspective, this separation manifests as a details container that makes the advanced filters collapsible. When closed, the Advanced section does not occupy any additional space and only indicates, via its header, that more options are available. Only when opened does a structured form unfold, which arranges the various filter dimensions into logically related groups. The concrete design begins with the configuration of the fields themselves:

javaCopy

codePart.setPlaceholder("e.g. ex-");codePart.setValueChangeMode(LAZY);codePart.setValueChangeTimeout(VALUE_CHANGE_TIMEOUT);codePart.addValueChangeListener(_->safeRefresh());urlPart.setPlaceholder("e.g. docs");urlPart.setValueChangeMode(LAZY);urlPart.setValueChangeTimeout(VALUE_CHANGE_TIMEOUT);urlPart.addValueChangeListener(_->safeRefresh());sortBy.setItems("createdAt","shortCode","originalUrl","expiresAt");dir.setItems("asc","desc");fromDate.setClearButtonVisible(true);toDate.setClearButtonVisible(true);fromTime.setStep(Duration.ofMinutes(15));toTime.setStep(Duration.ofMinutes(15));fromTime.setPlaceholder("hh:mm");toTime.setPlaceholder("hh:mm");

The text fields for the shortcode and URL provide meaningful placeholders and, like the global search, use a lazy ValueChange mode with a timeout. This prevents a new filter run from being triggered immediately with each input, while at the same time the filters respond quickly to changes. The sorting pairsortByis preassigned to you, with permissible values, and thus is embedded within a defined space of possible sorting strategies. Convenience functions supplement the date and time fields: Clear buttons, fixed 15-minute time grids, and placeholders for the time format help users enter data while reducing the risk of invalid values.

The spatial structure of Advanced Filters is designed to visually group related information. Instead of placing all components on a single long line, the implementation supports multiple upstream layouts. First, the date and time fields are merged into two groups:

kotlinCopy

varfromGroup=newHorizontalLayout(fromDate,fromTime);fromGroup.setDefaultVerticalComponentAlignment(Alignment.END);vartoGroup=newHorizontalLayout(toDate,toTime);toGroup.setDefaultVerticalComponentAlignment(Alignment.END);

The two horizontal layouts, „fromGroup“ and „toGroup“, ensure that date and time visually appear as a coherent unit. The vertical alignment at the bottom creates a calm, uniform appearance, even when field heights vary slightly. These groups are then embedded in a FormLayout, which forms the actual responsive structure:

javaCopy

FormLayoutsearchBlock=newFormLayout();searchBlock.setWidthFull();searchBlock.add(codePart,urlPart,newHorizontalLayout(codeCase,urlCase));searchBlock.add(fromGroup,toGroup);searchBlock.setResponsiveSteps(newFormLayout.ResponsiveStep("0",1),newFormLayout.ResponsiveStep("32rem",2),newFormLayout.ResponsiveStep("56rem",3));

Here, the textual filters – shortcode and URL – as well as the associated case sensitivity checkboxes are arranged together in a block. Below this are the groups for the time slot. ResponsiveSteps determines how many columns the layout may use at different widths. Below 32 rem, a single-column layout is selected; at medium width, two columns are available; and above 56 rem, the layout can be extended to three columns. In this way, the input mask remains readable and well-structured across wide desktop views and narrower windows or split screens.

The sorting control is deliberately visually decoupled from the search block, but positioned on the same horizontal axis. For this purpose, a separate toolbar area will be set up:

javaCopy

sortBy.setLabel(null);sortBy.setPlaceholder("Sort by");sortBy.setWidth("12rem");dir.setLabel(null);dir.setPlaceholder("Direction");dir.setWidth("8rem");HorizontalLayoutsortToolbar=newHorizontalLayout(sortBy,dir);sortToolbar.setAlignItems(FlexComponent.Alignment.END);

By removing the ComboBox labels and using placeholders, the interface remains compact without sacrificing intelligibility. The fixed width ensures stable alignment, while the horizontal grouping makes it clear that both fields functionally belong together. The alignment at the bottom blends with the rest of the header, where the filter boxes are also aligned on a common baseline.

Finally, the search block and sort bar are merged into a single header layout that constitutes the visible content of the Advanced area. This header layout is then embedded in aDetails component:

javaCopy

HorizontalLayoutadvHeader=newHorizontalLayout(searchBlock,sortToolbar);advHeader.setWidthFull();advHeader.setSpacing(true);advHeader.setAlignItems(FlexComponent.Alignment.START);advHeader.expand(searchBlock);advHeader.getStyle().set("flex-wrap","wrap");advHeader.setVerticalComponentAlignment(FlexComponent.Alignment.END,sortToolbar);advanced=newDetails("Advanced filters",advHeader);advanced.setOpened(false);advanced.getElement().getThemeList().add("filled");setSimpleSearchEnabled(!advanced.isOpened());

TheadvHeader ensures the search block occupies the available space, while the sorting tools are anchored to the right edge. Enablingflex-wrap allows the header to wrap to multiple lines within a limited width without disrupting the logical proximity of the elements. TheDetails component includes this header and makes the entire Advanced section collapsible. When closed, only the title “Advanced filters” remains visible; when opened, the complete form unfolds. The application of the filled theme also provides the area with a visual demarcation from the surrounding layout.

In terms of content, the Advanced Filters design prioritises making the essential dimensions of the data directly accessible. Shortcodes and destination URLs serve as textual entry points, and the search can be configured as case-sensitive or case-insensitive. The temporal context of the short URL – such as the creation or expiration interval considered – is mapped via combined date and time fields that explicitly work in the user’s local context. Finally, the sort field and sorting direction provide fine control over the order of displayed entries, enabling newly created or soon-expiring links to be brought to the foreground.

This expanded range of functions should not leave the user confronted with a confusing number of control elements. The precise spatial separation within the hinged container, the well-thought-out grouping of the fields, and the responsive arrangement therefore not only provide an aesthetic but, above all, a cognitive relief. The user can choose whether to use the standard global search options or switch to expert mode, which provides more detailed control over the database.

On this basis, the Advanced area can be seamlessly embedded into the rest of the architecture in the following chapters. In particular, the connection between the synchronisation logic and the refresh architecture described above demonstrates how the UI design and the internal state engine work in concert to keep even more complex filter requests stable and easy to understand.

Cheer Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-introduction-of-multiple-aliases-part-2/Fri, 12 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-introduction-of-multiple-aliases-part-2/Today’s update introduces another practical development step for the URL shortener. After the last few days were dedicated to UI refinement and the better structuring of detailed dialogues and form logic, the focus is now on an aspect that plays a significant role in the everyday life of many users: the flexible management of multiple aliases per target URL.<![CDATA[

Today’s update introduces another practical development step for the URL shortener. After the last few days were dedicated to UI refinement and the better structuring of detailed dialogues and form logic, the focus is now on an aspect that plays a significant role in the everyday life of many users: the flexible management of multiple aliases per target URL.

You can find the source code for this development status on GitHub underhttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-06

Here are the relevant screenshots of the Vaadin application.

What was previously only possible sequentially – an alias, a destination address – will now become a dynamic workspace. Users can add new aliases to existing short links, create variants for different use cases, or manage parallel campaigns without creating new records each time. The MultiAliasEditorStrict forms the core of this new workflow: inline validation, fast feedback, clear status indicators and an event flow that propagates changes directly into the OverviewView. The application responds immediately to user input and keeps the entire interface consistent, without requiring manual reloads or context switching.

But this is much more than just a UI comfort update. The increased flexibility in the alias structure inevitably leads to higher demands on server-side stability – especially on the stability of forwarders. Once several aliases point to the same destination address, the redirect logic must be deterministic and reliably detect erroneous requests. Therefore, a consistent, centralised request check was introduced in parallel with the UI extension to secure all forwarding operations and ensure predictable HTTP behaviour.

These changes combine two sides of the same coin: greater user freedom in their daily work and a robust technical foundation that reliably supports it. In the following, we will therefore take a closer look at the revised redirect implementation and the new security mechanisms that underpin stable, traceable, and maintainable redirect behaviour.

More Robust Redirects and HTTP Security

In parallel with the extensive improvements to the user interface, the server-side component of the URL shortener has also been revised. A central focus was on increasing the stability and security of HTTP communication. Especially when handling redirects, several methods have had to carry out similar checks – for example, whether the correct HTTP method was used or whether a request was incomplete. These repetitions led to inconsistent behaviour and made maintenance difficult.

In the process, a consistent request-handling system was introduced to centralise these checks. The code for the RedirectHandler clearly shows how a robust redirect is now implemented:

xmlCopy

package com.svenruppert.urlshortener.server.handler;import com.svenruppert.urlshortener.core.http.HttpStatus;import com.svenruppert.urlshortener.core.store.UrlMappingStore;import com.svenruppert.urlshortener.server.util.RequestMethodUtils;import com.svenruppert.urlshortener.server.util.ResponseWriter;import com.sun.net.httpserver.HttpExchange;import java.io.IOException;public class RedirectHandler implements HttpHandler, HasLogger { private final UrlMappingLookup store; public RedirectHandler(UrlMappingLookup store) { this.store = store; } @Override public void handle(HttpExchange exchange) throws IOException { if (! RequestMethodUtils.requireGet(exchange)) return; final String path = exchange.getRequestURI().getPath(); e.g. "/ABC123" if (path == null || !path.startsWith(PATH_REDIRECT)) { exchange.sendResponseHeaders(400, -1); return; } final String code = path.substring((PATH_REDIRECT).length()); if (code.isBlank()) { exchange.sendResponseHeaders(400, -1); return; } logger().info("looking for short code {}", code); Optional<String> target = store .findByShortCode(code) .map(ShortUrlMapping::originalUrl); if (target.isPresent()) { exchange.getResponseHeaders().add("Location", target.get()); exchange.sendResponseHeaders(302, -1); } else { exchange.sendResponseHeaders(404, -1); } }}

The RedirectHandler class ensures that only valid GET requests are accepted. Invalid methods – such as POST or DELETE – are caught by the RequestMethodUtils helper class. This prevents inadvertent writes to a read-only resource.

The check logic in RequestMethodUtils is deliberately kept simple and centralises all recurring security checks. The following excerpt shows how she works internally:

javaCopy

packagecom.svenruppert.urlshortener.server.util;importcom.svenruppert.urlshortener.core.http.HttpStatus;importcom.sun.net.httpserver.HttpExchange;importjava.io.IOException;publicclassRequestMethodUtils{publicstaticbooleanrequireGet(HttpExchangeexchange)throwsIOException{HasLogger.staticLogger().info("handle ... {} ",exchange.getRequestMethod());Objects.requireNonNull(exchange,"exchange");if(!" GET".equalsIgnoreCase(exchange.getRequestMethod())){writeJson(exchange,METHOD_NOT_ALLOWED,"Only GET is allowed for this endpoint.");returnfalse;}returntrue;}SNIP...}

This helper class serves as a compact security wall: every request-processing method calls it at the beginning. This creates predictable, consistent behaviour across the HTTP stack. This is particularly relevant for forwarding, as unauthorised or erroneous methods are reliably blocked.

Response generation is handled by ResponseWriter, which handles clean header configuration and uniform error output. In contrast to earlier implementations, which manually generated JSON or text responses in several places, there is now a central method:

javaCopy

publicstaticvoidwriteJson(HttpExchangeex,HttpStatushttpStatus,Stringmessage)throwsIOException{HasLogger.staticLogger().info("writeJson {}, {}",httpStatus,message);byte[]data=message.getBytes(UTF_8);Headersh=ex.getResponseHeaders();h.set(CONTENT_TYPE,JSON_CONTENT_TYPE);ex.sendResponseHeaders(httpStatus.code(),data.length);try(OutputStreamos=ex.getResponseBody()){os.write(data);}catch(Exceptione){HasLogger.staticLogger().info("writeJson {} (catch)",e.getMessage());byte[]body=("{\"error\":\""+e.getMessage()+"\"}").getBytes(StandardCharsets.UTF_8);try{ex.getResponseHeaders().set(CONTENT_TYPE,JSON_CONTENT_TYPE);ex.sendResponseHeaders(INTERNAL_SERVER_ERROR.code(),body.length);ex.getResponseBody().write(body);}catch(ExceptionignoredI){HasLogger.staticLogger().info("writeJson (catch - ignored I) {} ",ignoredI.getMessage());}}finally{try{ex.close();}catch(ExceptionignoredII){HasLogger.staticLogger().info("writeJson (catch - ignored II) {} ",ignoredII.getMessage());}}}

The merging of these helper classes yields a well-structured server architecture. Each handler is leaner, easier to read, and easier to test. For users, this results in a more reliable redirect: error messages are displayed consistently, HTTP status codes match the expected behaviour exactly, and unauthorised methods do not cause side effects.

This also gives the URL shortener additional robustness at the protocol level. Error handling is traceable, behaviour is transparent, and server components follow the same principles of clarity and reusability introduced in the UI. In this way, the system’s technical integrity is strengthened, as is user confidence in its stability.

Conclusion: User Comfort Meets Clean Design

With the completion of this development step, the URL shortener has developed into a much more mature system – both technically and conceptually. What started as a simple platform for creating and managing individual shortlinks has become a full-fledged application that balances user needs, security, and maintainability. The transition from a one-alias logic to a flexible multi-alias approach is probably the most visible progress. Still, the real strength of this update lies in the design’s consistent coherence.

From the user’s perspective, a tool is now available that makes workflows more fluid, consistent, and traceable. The detail-dialogue and the Create View form a harmonious pair: both use the same editor, the exact validation mechanisms and the same event flow. Actions are immediately visible, and changes are automatically displayed. The result is a surface that hides its technical complexity and instead focuses on efficiency and clarity.

A clear development is also noticeable on the architectural level. The interplay of Vaadin EventBus, component-based UI structure and centralised HTTP handling has resulted in a maintenance-friendly, extensible system. Every level – from the user interface to the validation logic to the server – follows the same principles: clear responsibilities, loose coupling, and transparent behaviour. This uniformity is evident not only in the code but above all in everyday use.

This step thus sets a clear benchmark for future project expansions. The concept of multiple aliases, the consistent reactivity of the UI, and the clean server design will not remain isolated features in the coming development phases; they will serve as a structural basis. Working on the user experience has shown that technical design and interaction quality are not mutually exclusive – on the contrary, they reinforce each other when they are consistently coordinated.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-introduction-of-multiple-aliases-part-1/Thu, 11 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-introduction-of-multiple-aliases-part-1/Introduction: More convenience for users With today’s development milestone for the URL Shortener project, a crucial improvement has been achieved, significantly enhancing the user experience. Up to this point, working with short URLs was functional, but in many ways it was still linear: each destination URL was assigned exactly one alias. This meant users had to create a new short URL for each context or campaign, even when the destination was identical. While this approach was simple, it wasn’t sufficiently flexible to meet real-world application requirements.<![CDATA[

Introduction: More convenience for users

With today’s development milestone for the URL Shortener project, a crucial improvement has been achieved, significantly enhancing the user experience. Up to this point, working with short URLs was functional, but in many ways it was still linear: each destination URL was assigned exactly one alias. This meant users had to create a new short URL for each context or campaign, even when the destination was identical. While this approach was simple, it wasn’t sufficiently flexible to meet real-world application requirements.

You can find the source code for this development status on Github underhttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-06

In everyday life, users quickly reach their limits when they want to map the same destination address for different purposes – for example, to analyse access from other sources or to separate internal and external uses. Working with only one alias per target URL forced them to work around and led to unnecessary redundancy in the stored mappings. Especially in professional environments where tracking, traceability, and reusability are essential, this situation was unsatisfactory.

With today’s development, this restriction has been lifted. The system now allows multiple aliases to point to a common destination URL. This focuses on user convenience: Existing short URLs can now be extended with new aliases without creating a new dataset. The process is accompanied by an intuitive user interface that integrates seamlessly into the application’s detailed dialogue.

This extension is more than just a functional improvement. It changes the way users interact with the system. Whereas previously each alias represented a separate entity, a central destination URL with variable identifiers is now introduced. The user no longer thinks in terms of individual links, but rather in terms of a flexible alias space mapped to a destination address. This reduces cognitive load and noticeably accelerates typical workflows.

On a technical level, this development was supported by the introduction of new UI components and event mechanisms that enable dynamic synchronisation between the detail dialogue and the overview. But the real meaning lies in the user experience itself: the shortener becomes a tool that adapts to real-world usage patterns – not the other way around.

From single alias to multi-alias management

The introduction of multi-alias management marks a clear turning point in the URL shortener’s development. While previously each short URL was inseparably bound to a single alias, the system now supports a more flexible, yet more natural, assignment model. From a technical standpoint, this means a structural expansion of the data model, but from the user’s perspective, it means one thing above all: freedom to manage one’s own short-link landscape.

At the heart of this innovation is the MultiAliasEditorStrict component, designed to intuitively guide users through managing multiple aliases. The editor lets you add new aliases, modify existing ones, or remove erroneous entries without opening modal dialogues or new dialogues. Instead, the user remains in a consistent context – the detailed view of a target URL – and can act directly there. This decision was deliberate to avoid friction in the interaction and maintain the flow between viewing and editing.

The implementation of MultiAliasEditorStrict follows a clear principle: strict validation with maximum user-friendliness. Each alias entered is immediately checked for empty values, duplicate entries, and syntactic correctness. Errors are visualised directly in the input field, providing the user with immediate feedback. This direct feedback builds trust in the input and prevents incomplete or erroneous data from reaching the persistence layer.

A key feature of this component is its interaction with the application’s event infrastructure. As soon as a new alias is saved or an existing alias is changed, the editor triggers an event that propagates across the entire UI. This means that other views – such as the overview list – are also automatically updated without requiring a reload. This event control ensures a reactive user interface that always reflects the current state.

From a technical perspective, multi-alias management illustrates how consistently implemented user orientation and clean architecture reinforce one another. The system remains modular, with components clearly separated, yet the user perceives a consistent, organic interface. What was previously a sequential process – create, check, correct – becomes an interactive dialogue with the system, which reacts to the user’s actions in real time.

Thus, MultiAliasEditorStrict is not only the heart of the new alias logic but also a harbinger of future developments. Its architecture lays the foundation for further interactions such as mass edits, alias groupings or time-controlled alias activations. In this sense, the component exemplifies the project’s evolution from a functional tool to a mature, user-centred application.

Source code and explanations

MultiAliasEditorStrict – Core of Multialias Management (excerpt)

xmlCopy

package com.svenruppert.urlshortener.ui.vaadin.components;SNIPPpublic class MultiAliasEditorStrict extends VerticalLayout { private static final String RX = "^[A-Za-z0-9_-]{3,64}$"; private final Grid<Row> grid = new Grid<>(Row.class, false); private final TextArea bulk = new TextArea("Aliases (comma/space/newline)"); private final Button insertBtn = new Button("Take over"); private final Button validateBtn = new Button("Validate all"); private final String baseUrl; private final Function<String,Boolean> isAliasFree; Server check (true = free) public MultiAliasEditorStrict(String baseUrl, Function<String,Boolean> isAliasFree) { this.baseUrl = baseUrl; this.isAliasFree = isAliasFree; build(); } ==== Public API for the parent view ==== public void validateAll() { var items = new ArrayList<>(grid.getListDataView().getItems().toList()); items.forEach(this::validateRow); grid.getDataProvider().refreshAll(); } public List<String> getValidAliases() { return grid .getListDataView() .getItems() .filter(r -> r.getStatus() == Status.VALID) .map(Row::getAlias) .collect(Collectors.toList()); } public void markSaved(String alias) { setStatus(alias, Status.SAVED, "saved"); } public void markError(String alias, String message) { setStatus(alias, Status.ERROR, (message == null ? "error" : message)); } public long countOpen() { return grid.getListDataView().getItems() .filter(r -> r.getStatus() != Status.SAVED).count(); } public void clearAllRows() { grid.setItems(new ArrayList<>()); } private void setStatus(String alias, Status s, String msg) { grid.getListDataView().getItems().forEach(r -> { if (Objects.equals(r.getAlias(), alias)) { r.setStatus(s); r.setMsg(msg); } }); grid.getDataProvider().refreshAll(); }}

Explanation. The component encapsulates all alias management within a standalone UI component. The constructor injects the baseUrl and a server-side availability-check function, keeping the editor testable and decoupling it from specific services. The public API provides the calling view with precise control points: validate all entries, extract valid aliases, set the status of individual rows to “SAVED” after successful saving, mark error states and count open works. The setStatus method updates the row status and triggers a UI refresh of the Grid DataProvider so that users can see the feedback immediately.

UIEvent for consistent refreshing

xmlCopy

import com.vaadin.flow.component.Component;import com.vaadin.flow.component.ComponentEvent;public class MappingCreatedOrChanged extends ComponentEvent<Component> { public MappingCreatedOrChanged(Component source) { super(source, false); }}

Explanation. This minimalist event is the link between the detail dialogue, create view and overview. After successfully saving new aliases, the UI fires this event; the OverviewView registers a listener and reloads its DataProvider. For the user, this means that changes are immediately visible without manual reloading.

Server-side redirect – consistent methodology control (excerpt)

javaCopy

@Overridepublicvoidhandle(HttpExchangeexchange)throwsIOException{if(!RequestMethodUtils.requireGet(exchange))return;// ... further code}

Explanation. Recurring checks and error handling are outsourced to utility methods. This reduces dispersion, prevents copy-and-paste divergences and increases the robustness of the routes. This principle is also reflected in the UI: validation, preparation, and persistence are clearly separated and orchestrated through well-defined interfaces.

Note on the presentation. In this chapter, only the excerpts that are decisive for user guidance are shown. The complete files, as well as the context of the private helper methods (build(), validateRow(…), bulk input parser, row POJO, and status enum), are located in thefeature/advent-2025-day-06 branch. The public API methods access them directly and map the validation-driven interaction flow.

Intelligent refresh of the overview

The integration of multi-alias management would be incomplete if changes to existing short links were not immediately visible in the overview. This is precisely where intelligent refresh comes in. The user should no longer be required to refresh the view after manually adding or changing aliases. Instead, the application automatically reacts to relevant events and ensures that the displayed data reflects the current state at all times.

Technically, this behaviour is implemented via Vaadin’s global EventBus. As soon as the MappingCreatedOrChanged event is triggered in the DetailsDialog, the OverviewView registers the signal and internally triggers the update process. The user thus experiences a seamless interaction between editing and overview. The logic responsible for this can be found directly in the constructor of the OverviewView:

javaCopy

ComponentUtil.addListener(UI.getCurrent(),MappingCreatedOrChanged.class,_->{logger().info("Received MappingCreatedOrChanged -> refreshing overview");refreshPageInfo();refresh();});

This code shows how the browser dynamically responds to changes in other UI components. The crucial part is registering the event listener with ComponentUtil.addListener. Every time a MappingCreatedOrChanged event is raised—for example, by the DetailsDialog after saving new aliases—the OverviewView receives it. The refreshPageInfo() and refresh() methods ensure that the displayed data is reloaded. As a result, status indicators, counters, and table contents always remain up to date without requiring user intervention.

Essential to this implementation is its simplicity and independence. The OverviewView does not know the source of the change. It only reacts to the generic event and fetches the data fresh from the server. This loosely coupled design prevents circular dependencies and makes the system robust to extensions. Future components that also trigger alias changes will be able to use the same event without requiring an adjustment to the overview.

It is also noteworthy that the EventBus system in Vaadin enables an apparent decoupling between UI elements. The DetailsDialog and the OverviewView share neither direct references nor mutual knowledge of their respective states. They communicate exclusively about events. This architecture is not only elegant, but also scalable. It enables future expansion of the system with additional listeners, such as notifications, logging, or statistics updates.

From the user’s perspective, this results in a reactive, immediate application experience. Changes to aliases or destination addresses are reflected in the overview without any noticeable delay. The user remains in the workflow and is always sure that what he sees reflects the current data state. This seemingly minor adjustment transforms the URL shortener’s operation into a modern, responsive interaction model that combines transparency and efficiency.

Improvements in the Create View

As part of this development step, the Create View also underwent a comprehensive overhaul. The aim was to adapt the process for creating new short URLs more closely to the updated user logic and, at the same time, to standardise workflows. While the detail dialogue is primarily responsible for maintaining existing mappings, the Create View serves as an entry point for new entries, now with the same convenience functions as the edit view.

The new version of the Create View offers a more transparent structure and uses a split layout to arrange form elements and preview information side by side. This allows the user to immediately see which aliases they are creating and how they relate to the destination URL. In addition, MultiAliasEditorStrict has been fully integrated, allowing multiple aliases to be captured directly, even when creating a new short URL.

xmlCopy

package com.svenruppert.urlshortener.ui.vaadin.views;SNIPP all importsimport static com.svenruppert.urlshortener.core.DefaultValues.SHORTCODE_BASE_URL;@Route(value = CreateView.PATH, layout = MainLayout.class)public class CreateView extends VerticalLayout implements HasLogger { public static final String PATH = "create"; private static final ZoneId ZONE = ZoneId.systemDefault(); private final URLShortenerClient urlShortenerClient = UrlShortenerClientFactory.newInstance(); private final TextField urlField = new TextField("Target URL"); private final DatePicker expiresDate = new DatePicker("Expires (date)"); private final TimePicker expiresTime = new TimePicker("Expires (time)"); private final Checkbox noExpiry = new Checkbox("No expiry"); public CreateView() { setSpacing(true); setPadding(true); setSizeFull(); urlField.setWidthFull(); Button saveAllButton = new Button("Save"); saveAllButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY); Button resetButton = new Button("Reset"); resetButton.addThemeVariants(ButtonVariant.LUMO_TERTIARY); configureExpiryFields(); FormLayout form = new FormLayout(); form.add(urlField, noExpiry, expiresDate, expiresTime); form.setResponsiveSteps( new FormLayout.ResponsiveStep("0", 1), new FormLayout.ResponsiveStep("900px", 2) ); form.setColspan(urlField, 2); HorizontalLayout actions = new HorizontalLayout(saveAllButton, resetButton); actions.setWidthFull(); actions.setJustifyContentMode(JustifyContentMode.START); Binder<ShortenRequest> binder = new Binder<>(ShortenRequest.class); binder.forField(urlField) .asRequired("URL must not be empty") .withValidator((String url, ValueContext _) -> { var res = UrlValidator.validate(url); return res.valid() ? ValidationResult.ok() : ValidationResult.error(res.message()); }) .bind(ShortenRequest::getUrl, ShortenRequest::setUrl); var editor = new MultiAliasEditorStrict( SHORTCODE_BASE_URL, alias -> { try { return urlShortenerClient.resolveShortcode(alias) == null; } catch (IOException e) { throw new RuntimeException(e); } } ); editor.setSizeFull(); editor.getStyle().set("padding", "var(--lumo-space-m)"); saveAllButton.addClickListener(_ -> { var validated = binder.validate(); if (validated.hasErrors()) return; if (!validateExpiryInFuture()) return; if (urlField.getValue() == null || urlField.getValue().isBlank()) { Notification.show("Target URL is empty", 2500, Notification.Position.TOP_CENTER); return; } editor.validateAll(); List<String> validAliases = editor.getValidAliases(); if (validAliases.isEmpty()) { Notification.show("No valid aliases to save", 2000, Notification.Position.TOP_CENTER); return; } Optional<Instant> expiresAt = computeExpiresAt(); int ok = 0; for (String alias : validAliases) { try { logger().info("try to save mapping {} / {} ", urlField.getValue(), alias); var customMapping = urlShortenerClient.createCustomMapping(alias, urlField.getValue(), expiresAt.orElse(null)); logger().info("created customMapping is {}", customMapping); if (customMapping != null) logger().info("saved - {}", customMapping); else logger().info("save failed for target {} with alias {}", urlField.getValue(), alias); editor.markSaved(alias); ok++; } catch (Exception ex) { editor.markError(alias, String.valueOf(ex.getMessage())); logger().info("failed to save URL with alias {}", alias); } } Notification.show("Saved: " + ok + " | Open: " + editor.countOpen(), 3500, Notification.Position.TOP_CENTER); }); resetButton.addClickListener(_ -> { clearFormAll(binder); editor.clearAllRows(); }); — SplitLayout var leftCol = new VerticalLayout(new H2("Create new short links"), form, actions); leftCol.setPadding(false); leftCol.setSpacing(true); leftCol.setSizeFull(); var rightCol = new VerticalLayout(new H2("Aliases"), editor); rightCol.setPadding(false); rightCol.setSpacing(true); rightCol.setSizeFull(); SplitLayout split = new SplitLayout(leftCol, rightCol); split.setSizeFull(); split.setSplitterPosition(40); add(split); }//... SNIPP private void clearFormAll(Binder<ShortenRequest> binder) { urlField.clear(); noExpiry.clear(); expiresDate.clear(); expiresTime.clear(); binder.setBean(new ShortenRequest()); urlField.setInvalid(false); }}

This implementation makes it clear that the Create-View is no longer just a simple input form, but now has the same range of functions as the detail dialogue. The user can capture multiple aliases, validate them directly, and save them with a single click. The integration of the MappingCreatedOrChanged event also ensures that the overview is automatically updated whenever a new mapping is created.

The split layout supports a clear visual separation between data entry and alias management. The user remains in the context of their input while also seeing how the selected aliases behave relative to the target URL. The reset function lets you reset inputs without reloading the page.

This revision made the Create View an integral part of the consistent user experience. It shares the validation and event logic with the detailed dialogue and thus serves as the foundation for all alias management. The user benefits from consistent behavior in both contexts, while the code remains clearer, more maintainable, and more extensible by reusing key components.

Consistent validation and error feedback

The growing complexity of the URL shortener’s user interface required validation logic that was both reliable and transparent. With the addition of multiple aliases, it became essential to provide precise feedback to the user when inputs are incomplete, duplicated, or syntactically incorrect. This feedback had to appear directly within the context of the respective component to avoid interrupting the interaction flow.

The validation system was therefore consistently integrated into Vaadin’s existing binder mechanism. The central entry point for this is the Create View, where the target URL field undergoes a strict validation. The implementation uses a dedicated UrlValidator that checks syntactic validity and schema compliance. The following excerpt from the Create View shows the binding and validation of the URL field:

xmlCopy

 Binder<ShortenRequest> binder = new Binder<>(ShortenRequest.class); binder.forField(urlField) .asRequired("URL must not be empty") .withValidator((String url, ValueContext _) -> { var res = UrlValidator.validate(url); return res.valid() ? ValidationResult.ok() : ValidationResult.error(res.message()); }) .bind(ShortenRequest::getUrl, ShortenRequest::setUrl);

This code demonstrates the interaction of mandatory field checking and semantic validation. The validator provides a precise error message for invalid inputs, which is displayed directly below the input field. The user can immediately see why an input was rejected. In this way, Vaadin’s classic form validation is supplemented with project-specific logic.

A similar principle is also used in MultiAliasEditorStrict, which checks each alias entry line by line. In addition to checking syntactic correctness, we also check whether an alias has already been assigned. Visual feedback is provided directly in the alias table, allowing the user to distinguish between valid, saved, and incorrect entries immediately. An excerpt from the corresponding class illustrates this mechanic:

javaCopy

publicvoidvalidateAll(){varitems=newArrayList<>(grid.getListDataView().getItems().toList());items.forEach(this::validateRow);grid.getDataProvider().refreshAll();}privatevoidvalidateRow(Rowr){vara=Objects.requireNonNullElse(r.getAlias(),"");if(!a.matches(RX)){r.setStatus(Status.INVALID_FORMAT);r.setMsg("3–64: A–Z a–z 0–9 - _");return;}longsame=grid.getListDataView().getItems().filter(x->x!=r&&Objects.equals(a,x.getAlias())).count();if(same>0){r.setStatus(Status.CONFLICT);r.setMsg("Duplicate in list");return;}if(isAliasFree!=null){try{if(!isAliasFree.apply(a)){r.setStatus(Status.CONFLICT);r.setMsg("Alias taken");return;}}catch(Exceptionex){r.setStatus(Status.ERROR);r.setMsg("Check failed");return;}}r.setStatus(Status.VALID);r.setMsg("");}

The validateAll() method iterates over all alias lines and calls an internal check for each. This validation is strict but transparent to the user: each alias line is given its own status indicator and text message. This means that error states are not collected as a list; they are displayed directly within the visual context. This form of inline validation complies with modern UI principles and minimises cognitive interruptions during data entry.

Another strength of this architecture is the standardisation of error feedback. Whether it’s an invalid URL, an empty alias, or a duplicate name, every discrepancy is handled the same way and communicated through the exact mechanism. This reduces inconsistencies and creates reliable feedback behaviour that users can intuitively understand. At the same time, the code remains easily extensible: new check rules can be added without modifying the existing logic.

The interaction between binder validation and alias-related status display thus creates a coherent feedback system. The user immediately learns which inputs have been accepted, which need correction, and when a record has been successfully saved. As a result, validation is no longer perceived as a hurdle, but as an integral part of an accurate and trustworthy user experience.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-from-grid-to-detail-understanding-the-user-experience-in-the-short-url-manager/Wed, 10 Dec 2025 10:32:12 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-from-grid-to-detail-understanding-the-user-experience-in-the-short-url-manager/The current UI from the user’s point of view On the first call, the user lands in the overview. The site is built on a Vaadin grid, whose header contains a search bar, paging controls, and a small settings button with a gear icon. The most essential flow begins with the table displaying immediately understandable columns: the shortcode as a clearly typographically separated monospace value with copy action, the original URL as a clickable link, a creation time in local format, and an expiration badge that visually communicates semantic states such as “Expired”, “Today” or “in n days” via theme colours. The whole thing is designed for quick viewing and efficient one-handed operation: a click on a data record opens the detailed dialogue if required; a right-click or the context menu offers direct quick actions; and the gear button can be used to show or hide visible columns live.<![CDATA[

The current UI from the user’s point of view

On the first call, the user lands in the overview. The site is built on a Vaadin grid, whose header contains a search bar, paging controls, and a small settings button with a gear icon. The most essential flow begins with the table displaying immediately understandable columns: the shortcode as a clearly typographically separated monospace value with copy action, the original URL as a clickable link, a creation time in local format, and an expiration badge that visually communicates semantic states such as “Expired”, “Today” or “in n days” via theme colours. The whole thing is designed for quick viewing and efficient one-handed operation: a click on a data record opens the detailed dialogue if required; a right-click or the context menu offers direct quick actions; and the gear button can be used to show or hide visible columns live.

The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-05

Central to everyday life is the small “Settings” button on the right of the search bar. It opens the column dialogue and directly affects the grid. This gives the user a lightweight tool to customise the table to their needs without losing context. In the dialogue, the user sees a tidy list of checkboxes—each named after the column header or the internal key. Each click immediately shows or hides the corresponding column; the state is persisted, so that when you revisit it, the view appears as if you had left. The behaviour is deliberately kept minimalist: no “Apply” button, but immediate feedback, supplemented by an “Apply bulk” option for collective changes.

The table is optimised for common workflows. The shortcode is formatted as a monospace span, and a copy button is right next to it that copies the full short URL to the clipboard. This eliminates the need to highlight text; a quick handover in chat, issue tracker, or email is done with one click. The original URL opens in a new tab, allowing the user to review the landing page without losing track. The creation and expiration times are kept compact; the expiration badge changes colour depending on the remaining term, signalling urgency.

javaCopy

grid.addComponentColumn(m->{varcode=newSpan(m.shortCode());code.getStyle().set("font-family","ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace");varcopy=newButton(newIcon(VaadinIcon.COPY));copy.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE);copy.getElement().setProperty("title","Copy ShortUrl");copy.addClickListener(_->{UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)",SHORTCODE_BASE_URL+m.shortCode());Notification.show("Shortcode copied");});...returnnewHorizontalLayout(code,copy,open,details);}).setHeader("Shortcode").setKey("shortcode").setAutoWidth(true).setResizable(true).setFlexGrow(0);grid.addColumn(m->DATE_TIME_FMT.format(m.createdAt())).setHeader("Created").setKey("created").setAutoWidth(true).setResizable(true).setSortable(true).setFlexGrow(0);grid.addComponentColumn(m->{varpill=newSpan(m.expiresAt().map(ts->{vardays=Duration.between(Instant.now(),ts).toDays();if(days<0)return"Expired";if(days==0)return"Today";return"in "+days+" days";}).orElse("No expiry"));pill.getElement().getThemeList().add("badge pill small");...returnpill;}).setHeader("Expires").setKey("expires").setAutoWidth(true);

In addition to the dialogue, there is a second, faster entry point via the context menu. Right-clicking on a line opens actions such as “Show details”, “Open URL”, “Copy shortcode” and “Delete…”. This is especially useful if the user already knows what they want to do with the current record without leaving the main view. The flow remains fluid because every action docks directly to the grid.

xmlCopy

GridContextMenu<ShortUrlMapping> menu = new GridContextMenu<>(grid);menu.addItem("Show details", e -> e.getItem().ifPresent(this::openDetailsDialog));menu.addItem("Open URL", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().open(m.originalUrl(), "_blank")));menu.addItem("Copy shortcode", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", m.shortCode())));menu.addItem("Delete...", e -> e.getItem().ifPresent(m -> confirmDelete(m.shortCode())));

If the user wants to know more or make changes, the application opens a standalone detail dialogue for the selected entry. From an interaction perspective, this is where deeper information and operations converge. The appeal remains deliberately unspectacular and fast:

javaCopy

privatevoidopenDetailsDialog(ShortUrlMappingitem){vardlg=newDetailsDialog(urlShortenerClient,item);dlg.addDeleteListener(ev->confirmDelete(ev.shortCode));dlg.addOpenListener(ev->logger().info("Open URL {}",ev.originalUrl));dlg.addCopyShortListener(ev->logger().info("Copied shortcode {}",ev.shortCode));dlg.addCopyUrlListener(ev->logger().info("Copied URL {}",ev.url));dlg.addSavedListener(_->refresh());dlg.open();}

As a result, the overview feels like a familiar workboard for the user, which remembers which columns really count, can be copied and opened quickly, and with context menus and a focused detail dialogue offers exactly the depth of interaction that is needed in everyday short URL life — no more, but also no less.

Detail dialogue on the data record

The detail dialogue is the central interaction element when users want to take a closer look at or edit a single entry in the system. It provides all relevant information about a shortened URL and enables basic operations, such as opening, copying, deleting, or saving the record. All features are designed to be available within the application without context switching.

The implementation of the dialogue in the project is in the DetailsDialog class. This class encompasses all the logic for displaying and manipulating aShortUrlMapping object. The dialogue is built entirely with Vaadin components:

xmlCopy

package com.svenruppert.urlshortener.ui.vaadin.views.overview;//SNIPPpublic class DetailsDialog extends Dialog implements HasLogger { private final URLShortenerClient urlShortenerClient; private final ShortUrlMapping mapping; private final DateTimeFormatter DATE_TIME_FMT= DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss") .withZone(ZoneId.systemDefault()); public DetailsDialog(URLShortenerClient urlShortenerClient, ShortUrlMapping mapping) { this.urlShortenerClient = urlShortenerClient; this.mapping = mapping; setHeaderTitle("Details for " + mapping.shortCode()); var form = new FormLayout(); var urlField = new TextField("Original URL"); urlField.setValue(Optional.ofNullable(mapping.originalUrl()).orElse("")); urlField.setWidthFull(); urlField.setReadOnly(true); var createdAt = new Span(DATE_TIME_FMT.format(mapping.createdAt())); var expiresAt = mapping.expiresAt() .map(ts -> { var days = Duration.between(Instant.now(), ts).toDays(); if (days< 0)return"Expired";if(days ==0)return"Today";return"in"+days+"days";}).orElse("Noexpiry");varexpiresField =newSpan(expiresAt);form.addFormItem(urlField,"OriginalURL");form.addFormItem(createdAt,"Created");form.addFormItem(expiresField,"Expires");varopenBtn =newButton(newIcon(VaadinIcon.EXTERNAL_LINK));openBtn.addThemeVariants(ButtonVariant.LUMO_TERTIARY);openBtn.getElement().setProperty("title","OpenoriginalURL");openBtn.addClickListener(_-> { UI.getCurrent().getPage().open(mapping.originalUrl(), "_blank"); fireEvent(new OpenEvent(this, mapping.originalUrl())); }); var copyShortBtn = new Button(new Icon(VaadinIcon.COPY)); copyShortBtn.addThemeVariants(ButtonVariant.LUMO_TERTIARY); copyShortBtn.getElement().setProperty("title", "Copy Short URL"); copyShortBtn.addClickListener(_ -> { UI.getCurrent().getPage() .executeJs("navigator.clipboard.writeText($0)", SHORTCODE_BASE_URL + mapping.shortCode()); Notification.show("Short URL copied"); fireEvent(new CopyShortEvent(this, mapping.shortCode())); }); var copyUrlBtn = new Button(new Icon(VaadinIcon.LINK)); copyUrlBtn.addThemeVariants(ButtonVariant.LUMO_TERTIARY); copyUrlBtn.getElement().setProperty("title", "Copy Original URL"); copyUrlBtn.addClickListener(_ -> { UI.getCurrent().getPage() .executeJs("navigator.clipboard.writeText($0)", mapping.originalUrl()); Notification.show("Original URL copied"); fireEvent(new CopyUrlEvent(this, mapping.originalUrl())); }); var deleteBtn = new Button(new Icon(VaadinIcon.TRASH)); deleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR, ButtonVariant.LUMO_TERTIARY); deleteBtn.getElement().setProperty("title", "Delete Short URL"); deleteBtn.addClickListener(_ -> { fireEvent(new DeleteEvent(this, mapping.shortCode())); }); var saveBtn = new Button("Save", _ -> { urlShortenerClient.update(mapping); fireEvent(new SavedEvent(this)); close(); }); var buttons = new HorizontalLayout(openBtn, copyShortBtn, copyUrlBtn, deleteBtn, saveBtn); add(form, buttons); } public Registration addOpenListener(ComponentEventListener<OpenEvent> listener) { return addListener(OpenEvent.class, listener); } public Registration addCopyShortListener(ComponentEventListener<CopyShortEvent> listener) { return addListener(CopyShortEvent.class, listener); } public Registration addCopyUrlListener(ComponentEventListener<CopyUrlEvent> listener) { return addListener(CopyUrlEvent.class, listener); } public Registration addDeleteListener(ComponentEventListener<DeleteEvent> listener) { return addListener(DeleteEvent.class, listener); } public Registration addSavedListener(ComponentEventListener<SavedEvent> listener) { return addListener(SavedEvent.class, listener); } public static class OpenEvent extends ComponentEvent<DetailsDialog> { private final String originalUrl; public OpenEvent(DetailsDialog src, String originalUrl) { super(src, false); this.originalUrl = originalUrl; } public String originalUrl() { return originalUrl; } } public static class CopyShortEvent extends ComponentEvent<DetailsDialog> { private final String shortCode; public CopyShortEvent(DetailsDialog src, String shortCode) { super(src, false); this.shortCode = shortCode; } public String shortCode() { return shortCode; } } public static class CopyUrlEvent extends ComponentEvent<DetailsDialog> { private final String url; public CopyUrlEvent(DetailsDialog src, String url) { super(src, false); this.url = url; } public String url() { return url; } } public static class DeleteEvent extends ComponentEvent<DetailsDialog> { private final String shortCode; public DeleteEvent(DetailsDialog src, String shortCode) { super(src, false); this.shortCode = shortCode; } public String shortCode() { return shortCode; } } public static class SavedEvent extends ComponentEvent<DetailsDialog> { public SavedEvent(DetailsDialog src) { super(src, false); } }}

The architecture follows a clear pattern: the dialogue displays data but does not trigger a direct UI update. Instead, it fires specific events (ComponentEvents) that are caught by the calling View (OverviewView). This means that the dialogue remains independent of its environment – a concept that can be implemented particularly elegantly in Vaadin.

The meaning of the individual buttons is immediately understandable: the open button opens the browser with the original URL, the two copy buttons copy the respective address to the clipboard, and the delete button sends a delete event that is further processed in the overview. It is particularly noteworthy that thesavebutton internallycalls urlShortenerClient.update(), thereby synchronising changes directly through the existing client-server infrastructure.

This makes the detail dialogue a self-contained UI element that encapsulates both presentation and interaction without dragging business logic into the interface. This pattern promotes reusability, testability, and a clear separation of responsibilities.

Context menu of grid rows

In the project, the context menu is anchored to theOverviewView and bound to theShortUrlMapping objects‘ grid. The source code shows how this menu is structured and what actions are offered in it:``

xmlCopy

GridContextMenu<ShortUrlMapping> menu = new GridContextMenu<>(grid);menu.addItem("Show details", e -> e.getItem().ifPresent(this::openDetailsDialog));menu.addItem("Open URL", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().open(m.originalUrl(), "_blank")));menu.addItem("Copy shortcode", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", m.shortCode())));menu.addItem("Delete...", e -> e.getItem().ifPresent(m -> confirmDelete(m.shortCode())));

The logic is simple but effective: each menu item performs a clearly defined action. These actions draw directly on the existing mechanisms of theOverviewView – such as theopenDetailsDialog() for detail views orconfirmDelete() for removing a data record.

Each menu action uses the methode.getItem().ifPresent(...)to ensure that a context object (that is, a selected entry in the grid) exists. This pattern protects the application from null references and makes the menu robust against unforeseen UI states.

The integration into theOverviewView is straightforward and follows Vaadin’s component-oriented approach. By binding to the grid, the menu is automatically linked to the respective rows. Users can thus interact with the data in a context-sensitive manner – an ergonomic advantage over classic toolbar or button solutions.

The connection to the detail dialogue is seamless: If the user selects “Show details” in the context menu, the DetailsDialog for the correspondingShortUrlMapping opens immediately. The interaction between the grid, context menu, and dialogue remains completely encapsulated within the view, keeping the code easy to maintain and understand.

Event integration between Overview and DetailDialog

The interaction between theOverviewView and theDetailsDialog is the functional heart of the editing workflow. It combines a tabular overview with a detailed view of individual datasets and ensures that changes in the dialogue are immediately reflected in the grid. The concept follows Vaadin’s component-oriented event mechanism, in which UI components can trigger their own events, which are processed by other elements – in this case, theOverviewView .

In theOverviewView, the dialogue opens when the user selects “Show details” from the context menu or double-clicks an entry. TheopenDetailsDialog() method takes over this task and, at the same time, binds all relevant listeners to the dialogue:

javaCopy

privatevoidopenDetailsDialog(ShortUrlMappingitem){vardlg=newDetailsDialog(urlShortenerClient,item);dlg.addDeleteListener(ev->confirmDelete(ev.shortCode));dlg.addOpenListener(ev->logger().info("Open URL {}",ev.originalUrl));dlg.addCopyShortListener(ev->logger().info("Copied shortcode {}",ev.shortCode));dlg.addCopyUrlListener(ev->logger().info("Copied URL {}",ev.url));dlg.addSavedListener(_->refresh());dlg.open();}

Each listener processes a specific event triggered by the dialogue itself. The trick is that theDetailsDialog defines these events as their own, type-safe subclasses ofComponentEvent. This allows theOverviewView torespond to actions in a targeted manner without knowing the dialogue’s implementation details. Examples includeDeleteEvent,CopyUrlEvent,CopyShortEvent,OpenEvent, andSavedEvent.

The crucial point is therefresh() at the end of the listener chain. This ensures that after a change to the dialogue, the grid data is reloaded. This keeps the display consistent and immediately reflects the current persistence state.

The dialogue itself triggers these events as soon as user actions occur. In the event of a deletion action, this is done via:

javaCopy

deleteBtn.addClickListener(_->{fireEvent(newDeleteEvent(this,mapping.shortCode()));});

Or when copying the short URL:

javaCopy

copyShortBtn.addClickListener(_->{UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)",SHORTCODE_BASE_URL+mapping.shortCode());Notification.show("Short URL copied");fireEvent(newCopyShortEvent(this,mapping.shortCode()));});

By triggering its own events, the dialogue can operate decoupled from its environment. It doesn’t know who calls it or what happens to the data afterwards – it just reports that an action occurred. This decoupling is an essential aspect of good UI architecture: it enables reuse and testable components without side effects.

TheOverviewView then takes responsibility for updating the system in response to these events. Exquisite is the use ofComponentEventListener, which has type safety and clean demarcation of the event classes. Vaadin thus offers a clearly structured and idiomatic way to model complex UI flows.

Another example shows the connection between the deletion process and the server communication. When aDeleteEvent is fired, theOverviewView callsthe internal methodconfirmDelete() to display a security prompt and, upon confirmation, delete the record server-side. Only then does the grid refresh again, so the removed entry disappears immediately.

This event-based integration ensures a consistent and reactive processing flow. Users experience changes in real time, without manual updates. For developers, the model offers a clear separation of responsibilities: theDetailsDialog encapsulates presentation and interaction, whileOverviewView orchestrates and synchronises them. This creates a UI structure that can be flexibly expanded and easily supplemented with new actions.

Validation and error feedback in the dialogue

A central element of the edit dialogue is input validation. This is to ensure that only correct and complete data is transmitted to the server. The focus is on checking URLs, as they form the core of everyShortUrlMapping.

In the current project, validation is handled using the VaadinBinder. The binder connects the UI fields to the properties of the underlying data model and provides built-in support for validations and error feedback.

A typical excerpt from the archive shows how this validation is implemented in practice:

javaCopy

binder.forField(urlField).asRequired("URL must not be empty").withValidator(url->{varvalidate=UrlValidator.validate(url);returnvalidate;},"Only HTTP(S) URLs allowed").bind(ShortUrlMapping::originalUrl,null);

Several steps are combined here:

1. Required field check –asRequired() ensures that the field is not left empty. If no input is made, Vaadin automatically displays an error message.
2. Content validation :withValidator() is also used to validate the URL format. This uses theUrlValidator helper class.

TheUrlValidator itself is a standalone utility class that syntactically checks whether a string is a valid HTTP or HTTPS URL. The implementation is:

javaCopy

packagecom.svenruppert.urlshortener.core.urlmapping;importjava.net.URI;publicfinalclassUrlValidator{privateUrlValidator(){}publicstaticbooleanvalidate(Stringurl){if(url==null||url.isBlank()){returnfalse;}try{varuri=URI.create(url);varscheme=uri.getScheme();returnscheme!=null&&(scheme.equalsIgnoreCase("http")||scheme.equalsIgnoreCase("https"));}catch(IllegalArgumentExceptione){returnfalse;}}}

This implementation is deliberately kept simple and uses only standard Java means. It verifies that the URL has a valid schema and starts with HTTP or HTTPS. This excludes all other protocols – an important aspect to avoid potential security risks (e.g. file:// or javascript://).

The result of this validation is immediately displayed in the UI. If the user enters an invalid URL, an error message will appear below the input field that comes from the message setwithValidator() (“Only HTTP(S) URLs allowed”). The Vaadin binder automatically controls this visual feedback.

The interaction amongBinder,TextField, andUrlValidator ensures consistent, user-friendly, and secure input validation. In addition, the system can be expanded if necessary, for example, to include additional checks (e.g. URL accessibility, blacklist filters or regex-based structure tests) without changing the existing architecture.

This validation ensures that theDetailsDialog is not just a display element, but also a reliable filter that catches erroneous input before it enters the persistence layer. This reduces error scenarios, increases data quality, and significantly improves the user experience.

UX fine-tuning and conclusion

The user experience (UX) of the interaction among theOverviewView, theDetailsDialog, and supporting UI components, such as theColumnVisibilityDialog, is the result of numerous targeted design decisions. The goal was to create a reactive yet simple interface that avoids visual overload while still offering complete control over the data.

The application consistently follows Vaadin’s guiding principles for a component-based UI. All interactions – from column selection to detail view – are encapsulated in clearly defined classes. The user benefits from an intuitive interaction logic: every action is available where it makes semantic sense, and every change is immediately visible.

An example of this UX approach is direct feedback on user actions. When a URL is copied or opened, a notification appears immediately:

javaCopy

Notification.show("Shortcode copied");

This micro-feedback improves the perception of the system response and confirms to the user that their action was successful. The same principle applies to several components, for example, when copying the original URL or saving it in theDetailsDialog.

Another part of the fine-tuning is the deliberate use of Vaadin theme variants to differentiate controls visually. For example, the delete button in the detail dialogue clearly signals its critical function through the combination of LUMO_ERRORandLUMO_TERTIARY:

javaCopy

deleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR,ButtonVariant.LUMO_TERTIARY);

This colouring adheres to the established Lumo design system and ensures that dangerous actions are immediately recognisable, without requiring additional explanatory text.

The modalization of the detail dialogue also contributes to user guidance:

javaCopy

setModal(true);setDraggable(true);setResizable(true);

This keeps the user’s focus clearly on the task at hand, while dragging and resizing offer flexibility. Especially in data-rich grids, this allows the dialogue to adapt to the situation without leaving the application.

Another UX aspect is maintaining the work context. After each action – such as saving or deleting – the grid is reloaded by therefresh() method without losing filter or paging information. This was deliberately kept this way in the implementation:

kotlinCopy

varreq=newUrlMappingListRequest(searchField.getValue(),pagination.getCurrentPage(),pagination.getPageSize());varmappings=urlShortenerClient.list(req);grid.setItems(mappings);

This means the user remains in the same position within the data view and maintains their orientation and workflow rhythm.

The OverviewView’s structure demonstrates how a consistent user experience can be achieved through small, targeted design decisions. Instead of complex UI frameworks, the application uses Vaadin’s native component toolkit and combines it with a clear event architecture and lightweight HTTP communication.

Result

The implementation of the editing workflow via the DetailsDialog shows how a modern, reactive application flow can be created with pure Vaadin Flow and Core Java– without external frameworks, reflection, or client-side JavaScript logic. The key lies in clean component decoupling, precise event control, and immediate user feedback.

The user receives a UI that adapts to their workflows: fast, comprehensible and error-resistant. At the same time, developers can create a maintainable architecture with clear responsibilities: the view orchestrates, the dialogue interacts, and the client synchronises. This interplay of simplicity and clarity is the actual UX fine-tuning of this implementation – and at the same time, the basis for the further expansion stages of the project.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-columnvisibilitydialog-part-2/Tue, 09 Dec 2025 09:55:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-columnvisibilitydialog-part-2/Server-Side Extension: PreferencesHandler and REST Interfaces The server-side extension for dynamic column visibility follows the same design logic as the UI: simplicity, clear accountability, and a precise data flow. While the OverviewView and the ColumnVisibilityDialog form the interface for the interaction, several specialized REST handlers handle the processing and persistence of the user settings. Their job is to process incoming JSON requests, validate them, translate them into domain operations, and return or store the current state.<![CDATA[

Server-Side Extension: PreferencesHandler and REST Interfaces

The server-side extension for dynamic column visibility follows the same design logic as the UI: simplicity, clear accountability, and a precise data flow. While the OverviewView and the ColumnVisibilityDialog form the interface for the interaction, several specialized REST handlers handle the processing and persistence of the user settings. Their job is to process incoming JSON requests, validate them, translate them into domain operations, and return or store the current state.

The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-04

Here’s the screenshot of the version we’re implementing now.

The central link is thePreferencesStore interface, which defines both read and write operations for column preferences. It is divided into two functional aspects – loading and updating:

javaCopy

publicinterfacePreferencesStoreextendsPreferencesLookup,PreferencesUpdater,HasLogger{}publicinterfacePreferencesLookup{Map<String,Boolean>load(StringuserId,StringviewId);}publicinterfacePreferencesUpdater{voidsaveColumnVisibilities(StringuserId,StringviewId,Map<String,Boolean>visibility);voiddelete(StringuserId,StringviewId,StringcolumnKey);voiddelete(StringuserId,StringviewId);voiddelete(StringuserId);}

This structure forms the foundation for the subsequent merchant classes. Each REST endpoint includes a clearly defined operation—loading, editing, or deleting column preferences. This creates a granular API that is idempotent on the one hand and enables precise error handling on the other.

The first port of call is theColumnVisibilityHandler, which accepts POST and DELETE requests. It is responsible for loading all of a user’s column settings within a view. The handler first checks that the request is complete, and then returns a flat JSON map that maps column names and visibility states:

javaCopy

@Overridepublicvoidhandle(HttpExchangeex)throwsIOException{switch(ex.getRequestMethod()){case"POST"->handleLoad(ex);case"DELETE"->handleDeleteAll(ex);case"OPTIONS"->allow(ex,"POST, DELETE, OPTIONS");default->methodNotAllowed(ex,"POST, DELETE, OPTIONS");}}privatevoidhandleLoad(HttpExchangeex)throwsIOException{varreq=fromJson(readBody(ex.getRequestBody()),ColumnInfoRequest.class);varvis=store.load(req.userId(),req.viewId());writeJson(ex,OK,toJson(vis==null?Map.of():vis));}

In addition, there are two specialized variants that separate the editing operations from each other. TheColumnVisibilitySingleHandler handles PUT and DELETE requests for individual columns. It takes a JSON object with user ID, view ID, and column name and updates the persistence accordingly:

javaCopy

privatevoidhandleSingleEdit(HttpExchangeex)throwsIOException{varreq=fromJson(readBody(ex.getRequestBody()),ColumnSingleEditRequest.class);varvisibility=Map.of(req.columnKey(),req.visible());store.saveColumnVisibilities(req.userId(),req.viewId(),visibility);writeJson(ex,OK,toJson(Map.of("status","ok")));}

For bulk uploads, theColumnVisibilityBulkHandler is used. This variant allows you to update multiple column states in a single request at the same time. The concept of bulk processing reduces latency and minimizes server-side write operations, for example, if the user changes many checkboxes in the dialog:

javaCopy

privatevoidhandleBulkEdit(HttpExchangeex)throwsIOException{varreq=fromJson(readBody(ex.getRequestBody()),ColumnEditRequest.class);store.saveColumnVisibilities(req.userId(),req.viewId(),req.changes());writeJson(ex,OK,toJson(Map.of("status","ok")));}

All three handlers follow the same design pattern: they validate the request, call the appropriate methods of thePreferencesStore and send back the correct HTTP status code. If successful, the server will respond with200 OK or204 No Content, and if the input is incorrect, it will respond with400 Bad Request. This convention makes the interface robust and easy to test.

The new API thus forms the backbone of the personalization logic. At the same time, it is an example of the loose coupling between UI and persistence: the client knows nothing about the storage strategy, the server nothing about the representation. Both communicate via well-defined JSON structures. This principle keeps the application extensible and independent of concrete implementation details – be it an in-memory store or long-term storage in EclipseStore.

The PreferencesClient: Round trip between UI and server

The connection between the user interface and persistence is established by the PreferencesClient. This component takes on the task of addressing the REST endpoints of the server side via HTTP, serializing data and converting the results into structures that can be used within the Vaadin UI. The client thus acts as an intermediary between the interaction logic of the interface and the data storage of the server – a classic link between display and state.

The structure of the PreferencesClient follows the established pattern of the project’s other service clients. It uses the Java standard library withHttpClient,HttpRequest andHttpResponse and completely dispenses with external dependencies. Communication takes place via clearly defined endpoints, all of which are documented in the class:

javaCopy

/** * Client for server-side column visibility preferences. * * Endpoints: * - POST /admin/preferences/columns -> load * - DELETE /admin/preferences/columns -> delete all (for a view) * - PUT /admin/preferences/columns/edit -> bulk edit * - PUT /admin/preferences/columns/single -> single edit */publicfinalclassColumnVisibilityClientimplementsHasLogger{

The client provides a set of methods that directly correspond to the server-side REST endpoints. The typical flow of a round trip starts with loading the stored visibility information:

javaCopy

publicMap<String,Boolean>load(StringuserId,StringviewId)throwsIOException,InterruptedException{varreqDto=newColumnInfoRequest(userId,viewId);varreq=requestBuilder(PATH_ADMIN_PREFERENCES_COLUMNS).POST(jsonBody(reqDto)).build();varresp=http.send(req,HttpResponse.BodyHandlers.ofString(UTF_8));if(resp.statusCode()==200){varbody=resp.body();if(body==null||body.isBlank())returnCollections.emptyMap();varparsed=parseJson(body);returnparsed.entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey,e->Boolean.parseBoolean(e.getValue())));}if(resp.statusCode()==204)returnCollections.emptyMap();thrownewIOException("Unexpected HTTP "+resp.statusCode()+" while loading column visibilities: "+resp.body());}

The method generates a simple JSON object from the user and view IDs and sends it to the server as a POST request. The answer contains a map of column names to truth values that can be used directly in the UI. Missing values are interpreted astrue , which is in line with the principle of full visibility.

The client offers two variants for changes: the editing of individual columns and the bulk update. While theeditSingle method specifically adjusts a column state,editBulk allows you to commit multiple changes in a single request. This separation corresponds to the semantic structure of the REST handlers:

javaCopy

publicvoideditSingle(StringuserId,StringviewId,StringcolumnKey,booleanvisible)throwsIOException,InterruptedException{varreqDto=newColumnSingleEditRequest(userId,viewId,columnKey,visible);varreq=requestBuilder(PATH_ADMIN_PREFERENCES_COLUMNS_SINGLE).PUT(jsonBody(reqDto)).build();varresp=http.send(req,HttpResponse.BodyHandlers.ofString(UTF_8));if(resp.statusCode()!=200){thrownewIOException("Unexpected HTTP "+resp.statusCode()+" on single edit: "+resp.body());}}

This method illustrates the principle of idempotency: multiple calls with the same data result in the same state without producing side effects. This is especially important for reactive user interfaces that update states asynchronously.

In combination with the ColumnVisibilityService, which acts as a wrapper, this creates a clear communication flow:

1. The UI interacts with the service via the ColumnVisibilityDialog.
2. The service calls the appropriate endpoints via the PreferencesClient.
3. The server side validates, stores and returns current states.
4. The service reflects the new values in the grid.

The entire system is therefore deterministic, comprehensible and fault-tolerant. If a transfer fails, the previous state is retained and the user can initiate the operation again. The PreferencesClient thus forms the technical basis for the stable persistence of visual preferences – a simple but extremely robust bridge between interaction and storage.

Persistence and EclipseStore integration

The storage of user preferences for column visibility is done in the same persistence system that was already introduced in the previous parts of the project: EclipseStore. This decision not only follows the consistency of the architecture design, but also underlines the goal of bringing together all system states – whether functional or visual – in a coherent data model. The persistence of user preferences thus becomes an equal part of the application.

The implementation of the PreferencesStore interface plays a central role in this. There are several concrete variants in the architecture: an in-memory version for tests and volatile scenarios as well as an EclipseStore-supported version for productive operation. Both follow the same contract model, but differ in the underlying storage medium.

The following excerpt shows the interface structure on which the implementations are based:

javaCopy

com.svenruppert.urlshortener.api.store.preferences.PreferencesStorepublicinterfacePreferencesStoreextendsPreferencesLookup,PreferencesUpdater,HasLogger{}com.svenruppert.urlshortener.api.store.preferences.PreferencesLookuppublicinterfacePreferencesLookup{Map<String,Boolean>load(StringuserId,StringviewId);}com.svenruppert.urlshortener.api.store.preferences.PreferencesUpdaterpublicinterfacePreferencesUpdater{voidsaveColumnVisibilities(StringuserId,StringviewId,Map<String,Boolean>visibility);voiddelete(StringuserId,StringviewId,StringcolumnKey);voiddelete(StringuserId,StringviewId);voiddelete(StringuserId);}

These clear contracts form the basis for various storage strategies. In particular, the EclipseStore variant (EclipsePreferencesStore) integrates seamlessly into the existing object graph. When saving, it checks whether an entry for the combination of user and view ID already exists and updates the visibility information accordingly. A new entry is only created if no previous state exists. This logic ensures idempotency and prevents unnecessary duplicates.

The following fragment from the EclipseStore implementation illustrates the principle:

javaCopy

com.svenruppert.urlshortener.api.store.provider.eclipsestore.patitions.EclipsePreferencesStore(simplifiedexcerpt)@OverridepublicvoidsaveColumnVisibilities(StringuserId,StringviewId,Map<String,Boolean>visibility){varuserPrefs=dataRoot.preferences().computeIfAbsent(userId,_->newHashMap<>());varviewPrefs=userPrefs.computeIfAbsent(viewId,_->newHashMap<>());viewPrefs.putAll(visibility);storage.storeRoot(dataRoot);}

This method illustrates the simplicity and directness of persistence: All preferences are stored in the root object, so they automatically become part of the transactional storage in the EclipseStore. The entire system benefits from object persistence without classic database tables or ORM layers.

A key feature of this solution is its resilience. Because EclipseStore automatically synchronizes changes, user preferences are reliably maintained even if the system terminates unexpectedly. In addition, the application benefits from the direct object addressing that EclipseStore offers: no complicated ORM mappings are necessary, and changes to the data structure are automatically applied. Storage thus remains as natural as the modeling itself.

In addition, the persistence layer remains extensible. By separating the interface and implementation, alternative storage mechanisms can also be used in the future – such as an encrypted file variant for particularly sensitive environments or a network-based solution for multi-user systems. The existing code of the UI and the server handler would not have to be adapted for this, as all access takes place via thePreferencesStore interface.

With this integration, the persistence path is complete: from the user to the dialog, the service, the client, the REST interfaces and finally to the EclipseStore, the data flow is consistently typed and consistent. This finally makes column visibility a part of the permanent system state – a visible sign that user interaction and data storage are no longer separate worlds in this architecture, but two perspectives on the same, persistent context.

Architecture and Event Flow

The introduction of dynamic column visibility has not only spawned new UI and server components, but has also refined the entirety of the application architecture. It has created a consistent flow of events that aligns all levels, from the user interface down to persistence. This consistency makes the function not only robust, but also expandable and testable.

At the center of this flow is the OverviewView as the trigger for user interaction. When the user presses the gear icon, theColumnVisibilityDialogopens, which in turn communicates via theColumnVisibilityService. This service calls theColumnVisibilityClient, which in turn calls the server’s REST endpoints. The server processes the request via the appropriate handlers – such asColumnVisibilityHandler,ColumnVisibilitySingleHandler orColumnVisibilityBulkHandler – and writes the changes to the EclipseStore via thePreferencesStore. Finally, the return channel ensures that the stored visibility is automatically restored during the next initialization.

The sequence can be schematically represented as follows:

javaCopy

Users→OverviewView→ColumnVisibilityDialog→ColumnVisibilityService→ColumnVisibilityClient→RESTAPI→PreferencesHandler→PreferencesStore→EclipseStore→PersistentStorage

Each of these components fulfils a clearly defined role and communicates exclusively via well-defined interfaces. In this way, the architecture strictly follows the principle of separation of concerns. The UI part remains completely decoupled from persistence, while the server logic does not require any knowledge of the frontend. Changes to one layer do not have an immediate effect on the other layers.

A particularly elegant aspect can be seen in the interaction between the user interface and the event system. The OverviewView is connected to theStoreEvents event system . As soon as changes to the database are detected on the server side, a signal is sent to the UI via a publish/subscribe pattern. The application responds to this with a synchronized refresh:

javaCopy

subscription=StoreEvents.subscribe(_->getUI().ifPresent(ui->ui.access(this::refresh)));

This mechanic, which is already known from the previous days of the Advent calendar, ensures that the user interface and saved state always match. It forms the foundation of reactivity throughout the project.

The event flow within the application is thus bidirectional: On the one hand, the user initiates interactions that lead through the round trip to persistence. On the other hand, persistent changes can in turn have an effect on the UI. This interplay between action and reaction creates a dynamic coherence that makes the system tangibly alive.

Compared to classic web applications, in which the state between client and server is often fragmented or only temporary, this architecture establishes a continuous data cycle. Every step is traceable and typified – from the checkbox in the dialog to the saved Boolean map in the EclipseStore. This minimizes sources of error and significantly increases the maintainability of the application.

All in all, it can be seen that the addition of dynamic column visibility is not just a new function, but a maturation of the entire system architecture. It combines UI, event control and persistence into a coherent whole and thus lays the foundation for further personalization mechanisms based on the same principles.

UX and ergonomics: Self-determined work

With the introduction of dynamic column visibility, the relationship between user and application is fundamentally changing. While the previous functions were primarily aimed at consistency, stability and clarity, one aspect is now coming to the fore that was previously mainly implicit: the self-determination of the user. The OverviewView interface is no longer a rigid reflection of developers’ decisions, but becomes a customizable tool that is subordinate to the individual ways of working of its users.

This change is already evident in the way the dialogue has been shaped. TheColumnVisibilityDialog follows the principle of minimal friction: the user should be able to adjust his view without losing context or feeling cognitive load. Therefore, the dialog opens modally, focuses on the current task and offers only those controls that are relevant for the decision. Each checkbox represents a column – nothing more, nothing less. The immediate feedback in the grid after each change ensures that the user experiences the effects of their decision directly.

The combination of immediate feedback and persistent storage has a psychologically important effect: it creates trust. When a system visibly responds to the user’s preferences and restores them unchanged the next time it is opened, the feeling of stability and control is created. This is one of the cornerstones of good interaction design – the system should not only work, but also appear reliable.

The placement of the function on the surface also follows ergonomic considerations. The gear icon that opens the dialog is deliberately designed to be unobtrusive. It is accessible at all times, but not dominant. This keeps the focus on the content while keeping control over the presentation always within reach. The user decides for himself when and to what extent he makes adjustments. The system does not impose itself, but reacts.

Combined with the automatic restoration of preferences at launch, it creates a gentle personalization that feels organic. The application not only remembers states – it also remembers habits. If you have hidden certain columns, you signal a personal usage pattern that will be quietly respected on your next visit. This form of implicit personalization is efficient because it does not require additional dialogues, profile settings or user accounts and still creates an individual experience.

The developer perspective is not left out. The architecture was designed in such a way that the expansion of further personal settings – such as sort orders, column widths or layout preferences – would be possible seamlessly. The existing concept of thePreferencesStore allows the inclusion of additional parameters without structural changes. This creates an extensible system that may seem simple in the user experience, but is highly modular in the background.

This makes the dynamic column visibility feature more than just a convenience feature. It is an expression of a philosophy: the user should be able to find himself in the application. Software that adapts instead of demanding adaptation creates acceptance and productivity at the same time. The user remains the focus, technology takes a back seat – exactly where it has its greatest effect.

Technical reflection and safety aspects

The introduction of dynamic column visibility is not only a functional extension, but also a technical statement. It shows that personalization in a modern web application does not have to be at odds with robustness, security, and maintainability. Rather, individualization can be integrated as a controlled and comprehensible extension of the existing security model.

From a technical point of view, the new function touches on several security-relevant levels. First of all, the handling of user IDs should be mentioned. In the current implementation, the username “admin” serves as a placeholder, but in a production environment, this information comes from an authenticated context. It is important that the preferences are strictly tied to the authenticated user in order to ensure a clear delimitation of the visibility areas. A user may only access their own stored preferences – never anyone else’s.

The interfaces themselves are designed in such a way that they already implicitly assume this separation. Each request contains both auserId and aviewId. The REST handlers check that both values are present and valid. Missing or empty fields will result in a controlled rejection of the request:

javaCopy

if(isBlank(req.userId())||isBlank(req.viewId())){writeJson(ex,BAD_REQUEST,"userId and viewId required");return;}

This simple but effective validation protects against non-specific calls and prevents unwanted manipulation of the database. In combination with the REST semantics, which only allow idempotent operations (POST, PUT, DELETE), it ensures that the application behaves deterministically even under heavy load or during repeated requests.

Another important aspect concerns data integrity within thePreferencesStore. Since the stored values are organized in a map<string, Boolean> the question arises as to how to deal with invalid or manipulated data. The implementation uses type checking and defensive programming for this purpose: Only known column names are adopted, unknown or incorrect keys are discarded. This keeps the persisted state consistent, even if a failed client sends data.

javaCopy

varknownKeys=grid.getColumns().stream().map(Grid.Column::getKey).filter(Objects::nonNull).toList();Map<String,Boolean>state=service.mergeWithDefaults(knownKeys);

This logic in the UI prevents unauthorized or inappropriate values from being stored in the first place. The dialog works exclusively with those columns that actually exist in the grid, and is therefore inherently resistant to manipulation attempts on the client side.

In addition to functional safety, resilience to errors also plays an essential role. All communication paths between client and server are embedded intry-catch blocks. If the connection is lost or the server does not respond, the application will continue to be usable. The service reports errors via logging without blocking user interaction:

javaCopy

try{client.editBulk(userId,viewId,changes);}catch(IOException|InterruptedExceptione){logger().warn("Persist bulk failed {}: {}",changes.keySet(),e.toString());}

This approach ensures that server communication failures do not have a negative impact on the user experience. The user can continue working; its changes are automatically synchronized on the next successful connection.

Overall, it can be seen that the security concept of the application has not been weakened by the expansion, but strengthened. The integrity of the system is maintained through clear interfaces, typed data structures, defensive programming and strict separation of user contexts. The dynamic column visibility proves that user freedom and system security are not mutually exclusive, but – if implemented carefully – reinforce each other.

Result

With the introduction of dynamic column visibility, the development of the administration interface has reached a decisive milestone. What began as a static, system-centric overview in the first days of the Advent calendar has now developed into an interactive, user-centered platform. The application learns, reacts and remembers – it becomes a tool that adapts to the user instead of forcing him to get used to its structures.

Instead of replacing existing components, they have been expanded and precisely linked to each other. The newColumnVisibilityDialog fits seamlessly into the existing UI concept, thePreferencesClient uses the same mechanisms as the previous REST clients, and persistence via theEclipseStore closes the circle of dataflows. Everything interlocks, without breaks or special paths. This homogeneity is not a coincidence, but the result of an architecture that has anchored openness and coherence as fundamental principles from the very beginning.

From a user experience perspective, the extension is a step towards self-determination. Users can now personalize their workspace without having to struggle through menus and options. The interface remains light, the behavior is comprehensible and the stored preferences create trust in the stability of the application. The system reacts to user decisions instead of forcing them – a paradigm shift that noticeably increases the quality of interaction.

The clear separation between user context, visibility logic and data storage prevents manipulation and ensures data integrity. The use of typed maps and well-defined REST endpoints minimizes sources of error. Persistence viaEclipseStore also offers the advantage that the application remains consistent at all times, even in the event of abrupt interruptions or partial failures.

Looking ahead opens up new possibilities. On the basis of the now established preference system, further personalization dimensions can be implemented: column sequences, sorting preferences, color schemes or layout options. By loosely coupling UI, service and persistence, the system is ready for these extensions without having to change existing structures. Integration with user profiles or role-based access models could also be built on it.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-columnvisibilitydialog-part-1/Mon, 08 Dec 2025 22:42:05 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-columnvisibilitydialog-part-1/From observer to designer: User control at a glance With the fourth day of the Advent calendar, the perspective on the application changes fundamentally. While in the previous expansion stages users reacted primarily to prepared structures, an element of active design is now being added. The “Overview” has so far been a central mirror of the system state: It showed all saved short links, allowed them to be filtered and provided an in-depth insight into individual objects with the detailed dialog. But the structure and visibility of the columns were immutable up to this point. The user always saw the selection that the developer had defined as meaningful. With the introduction of dynamic column visibility, this state is removed.<![CDATA[

From observer to designer: User control at a glance

With the fourth day of the Advent calendar, the perspective on the application changes fundamentally. While in the previous expansion stages users reacted primarily to prepared structures, an element of active design is now being added. The “Overview” has so far been a central mirror of the system state: It showed all saved short links, allowed them to be filtered and provided an in-depth insight into individual objects with the detailed dialog. But the structure and visibility of the columns were immutable up to this point. The user always saw the selection that the developer had defined as meaningful. With the introduction of dynamic column visibility, this state is removed.

1. From observer to designer: User control at a glance
2. Concept: Visibility as a user preference
3. The new UI interaction: ColumnVisibilityDialog
4. Integration with the OverviewView

The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-04

Here’s the screenshot of the version we’re implementing now.

The grid in the OverviewView is thus transformed from a static display element into a customizable tool. Users can choose which columns are visible and which are hidden. This step seems minor at first glance, but it leads to a new form of interaction between user and application. The system no longer determines the surface structure, but reacts to individual preferences. The application enters into a dialogue with the user, in which the view of data becomes part of the personalization.

This extension is a direct continuation of the basis laid in the previous days. The filter and search mechanisms from the first part have taken control of the data flow. The integration of EclipseStore has made the state visible and at the same time established permanent storage in the system. Finally, the detail dialog created the separation between the overview and the object context. With this in mind, the new feature is the next logical step: it shifts control over the presentation of data from the system to the user without complicating the underlying technical structure.

From a user experience perspective, this change is significant. Those who work with the administration interface on a daily basis develop their own rhythm in dealing with data. Some users focus on expiring links, others on manual aliases, or on the traffic of their short links. For all these scenarios, a static column arrangement is a hindrance. The ability to hide irrelevant information and tailor the view to your own work processes creates not only convenience, but also efficiency. The user thus turns from a passive observer into an active designer of his or her work environment.

The implementation of this principle in Vaadin is based on a conscious balance between simplicity and persistence. On the one hand, the function should be immediately available: One click opens a dialog, a handful of checkboxes controls the visibility of the columns, and the result becomes immediately visible. On the other hand, these attitudes should not remain fleeting. The selected configuration is permanently saved and automatically restored on each reboot. This persistence in visual behavior takes up the idea of the visible system state and transfers it to the user level.

The introduction of dynamic column visibility thus marks the beginning of a new chapter in the development of the administration interface: It is no longer just about the presentation of data, but about the design of one’s own work context. This feature marks a paradigm shift in the UI design of the project. Instead of predefined structures, a flexible, learning interface is created that adapts to individual ways of working and thus takes a decisive step towards true user-centricity.

Concept: Visibility as a user preference

The introduction of dynamic column visibility not only marks an ergonomic improvement in the user interface, but also a conceptual expansion of the application architecture. For the first time, an aspect of user interaction is permanently personalized – regardless of the session or system state. This innovation requires a separate model for preferences that works consistently on both the client and server side. Visibility thus becomes a stored property that is mediated between the user and the application.

The central idea is that each view in the application has its own identity. For the OverviewView, this means that its configuration – such as which columns are visible – is clearly stored under a view ID. At the same time, each user instance is identified, for example by means of a user ID, and treated as a carrier of individual preferences. These two dimensions – user and view – are key to storing visibility information. The concept thus follows a clear scheme: a user ID, a view ID and an assignment of column names to truth values.

The introduction of this logic required a new interface between the UI and the backend. The application had to learn to deal with semantic information about user decisions, not just data objects. On the server side, this is done by the new PreferencesHandler, which serves as a REST endpoint and allows both loading and saving the settings. On the client side, the new PreferencesClient component communicates with this handler and manages the transfer of the data structures. The connecting format is deliberately simple: a map<string, Boolean> in which each key corresponds to the column name and the value describes the visibility. This clarity makes it possible to keep the mechanism generic and later to extend it to other areas of the application.

The decision to model preferences as a separate entity is an important architectural step. It separates application data—such as short links and expiration information—from the presentation data that drives the user’s behavior within the interface. In this way, the persistence layer is expanded by a new category: no longer only technical but also contextual states. This pattern is reminiscent of the separation between domain logic and UI logic, which is central to the clean architecture approach, and consistently transfers it to the behavior of the user interface.

Another goal of the implementation is transparency. The preferences should not act as a black box, but remain traceable at all times. Changes to the column configuration are therefore immediately visible and saved at the same time. The user does not feel a break between the interaction and the result, and yet their behavior is preserved permanently. This dual nature – volatile response and persistent storage – makes the system a learning tool. The application remembers how its users work and automatically reproduces this behavior on the next call.

This adds a semantic layer to the UI concept: visibility is no longer understood as a technical property of the grid component, but as an expression of an individual way of working. The application itself becomes a customizable medium that adapts to people, not the other way around. In this perspective, the dynamics of column visibility become a symbol for the transition from standardized operation to personalized control – a principle that will also be reflected in the technical implementation in the following chapters.

The new UI interaction: ColumnVisibilityDialog

With the introduction of dynamic column visibility, the application’s user interface gets a new level of interaction. The previously immutable structure of the OverviewView is extended by a component that allows the user to directly influence the structure of his working environment. At the heart of this is theColumnVisibilityDialog – a dialog box that lists all available columns in the overview table and controls their visibility via simple checkboxes. It embodies the idea of a configurable but intuitive interface.

The concept of dialogue follows a deliberate reduction: no cluttered settings menu, no hidden options, but a clear, focused place for a single task. When the dialog is opened, the application reads the saved preferences of the current user via thePreferencesClient and sets the checkboxes according to the stored values. Each checkbox represents a column of the OverviewView – such as “Shortcode”, “URL”, “Created” or “Expires”. Changes are immediately visible: As soon as the user selects or deselect a checkbox, the corresponding column in the grid is shown or hidden. This creates a direct interplay between action and reaction, which strengthens the perception of control.

The ColumnVisibilityDialog is deliberately implemented as an independent component. It extends the Vaadin class Dialog and follows the same architectural philosophy as the already known DetailsDialog from the previous days. This parallel allows it to fit seamlessly into the existing structure: it has a clear lifecycle logic, its own events and a minimalist layout that is designed for comprehensibility and efficiency. The dialog typically contains a vertical list of checkboxes that is dynamically generated from the grid’s column configuration. A save button permanently updates preferences through the PreferencesClient, while a “Reset” button restores the original default view.

From a technical point of view, the dialog is based on a simple but elegant data flow. When opened, the client loads a map<string, Boolean> whose keys correspond to the column identifiers. This map is converted into UI elements, with the current visibility mirrored directly in the grid. Clicking on “Save” triggers the methodsaveColumnVisibilities(userId, viewId, visibilityMap), which transmits the changed settings to the server. The JSON format is deliberately left flat to be both human-readable and easily testable. Server-side storage in EclipseStore ensures that these changes are not only retained for the current session, but also permanently.

The interplay of dialog, grid and client shows the strength of the modular approach. The ColumnVisibilityDialog does not know any details about persistence or grid implementation – it works exclusively via clearly defined interfaces. This keeps the component independent, reusable, and easily expandable. This pattern is particularly important in the Vaadin architecture because it ensures the loose coupling between user interaction and system health.

From a user experience perspective, the dialog performs two tasks at the same time. It provides control without complexity and creates confidence in the stability of the system. Users experience that their individual decisions are not only temporary, but are structurally anchored in the application. Every adjustment is reproducible, every change is reversible. This creates a natural balance between freedom and security, which reflects the character of the entire project. The ColumnVisibilityDialog thus becomes a visible expression of the idea that an administration interface should not only provide tools, but also adapt to the way its users work.

To illustrate this, the following is an original excerpt from the dialog, which shows the coupling between checkboxes and grid columns as well as the collection of changes for an optional collective persistence. The code works directly on the column keys of the grid and reflects state changes back to the interface without any detours.

javaCopy

Map<String,Boolean>pending=newLinkedHashMap<>();grid.getColumns().forEach(col->{finalStringkey=col.getKey();if(key==null)return;onlyaddressablecolumnsbooleanvisible=state.getOrDefault(key,true);col.setVisible(visible);varcb=newCheckbox(col.getHeaderText()!=null?col.getHeaderText():key,visible);cb.addValueChangeListener(ev->{booleanv=Boolean.TRUE.equals(ev.getValue());col.setVisible(v);pending.put(key,v);});form.add(cb);});varbtnApply=newButton("Apply bulk",_->{if(!pending.isEmpty()){service.setBulk(newLinkedHashMap<>(pending));pending.clear();}close();});btnApply.addThemeVariants(ButtonVariant.LUMO_PRIMARY);

The dialog deliberately delegates the actual persistence to the service. This node abstracts the transport and writes the changed visibilities to the server in one go. The following fragment shows the corresponding method in the service, which encapsulates the round trip and at the same time remains fault-tolerant.

javaCopy

publicvoidsetBulk(Map<String,Boolean>changes){if(changes==null||changes.isEmpty())return;try{client.editBulk(userId,viewId,changes);}catch(IOException|InterruptedExceptione){logger().warn("Persist bulk failed {}: {}",changes.keySet(),e.toString());}}

In this way, the ColumnVisibilityDialog remains lean and focused on the interaction, while the persistence logic is handled centrally via the service and the underlying client. This division of responsibilities ensures that the user experience feels clear and responsive without compromising the technical integrity of the application.

Integration with the OverviewView

The introduction of the ColumnVisibilityDialog would have remained incomplete if it had not been organically embedded in the existing OverviewView. This is exactly where the strength of the architecture becomes apparent, which was designed for loose coupling and clear responsibilities from the very beginning. The OverviewView serves as the application’s central window—connecting data, filters, paging, and interaction. Now it is expanding this spectrum to include a persistent personalization dimension.

At the center of the integration is an inconspicuous but meaningful button in the search bar: the gear icon. This icon acts as an entry point into the configuration logic. The decision not to hide the dialog in a menu or in a separate view follows the idea of control available at all times. Users should be able to customize the presentation of the data exactly where they look at it. The code snippet from the OverviewView illustrates this direct integration:

javaCopy

btnSettings.addThemeVariants(ButtonVariant.LUMO_TERTIARY);btnSettings.getElement().setProperty("title","Column visibility");btnSettings.addClickListener(_->newColumnVisibilityDialog<>(grid,columnVisibilityService).open());

With just a few lines, the entire functionality is connected. The button opens a new instance of the dialog, which receives the current grid and the corresponding service instance. This service instance manages all operations on the stored preferences. In this way, a clear data flow is created: the OverviewView calls the dialog, the dialog interacts with the ColumnVisibilityService, and the ColumnVisibilityService in turn communicates with the server via the ColumnVisibilityClient.

The initialization of the service takes place when the view is attached to the UI. This phase is ideal for loading user contexts and creating initial states. The relevant code block from theonAttach method shows this flow:

javaCopy

@OverrideprotectedvoidonAttach(AttachEventattachEvent){super.onAttach(attachEvent);this.columnVisibilityService=newColumnVisibilityService(columnVisibilityClient,"admin","overview");varkeys=grid.getColumns().stream().map(Grid.Column::getKey).filter(Objects::nonNull).toList();varvis=columnVisibilityService.mergeWithDefaults(keys);grid.getColumns().forEach(c->{vark=c.getKey();if(k!=null)c.setVisible(vis.getOrDefault(k,true));});refresh();subscription=StoreEvents.subscribe(_->getUI().ifPresent(ui->ui.access(this::refresh)));}

When setting up the view, the service is first created and linked to the user and view ID. All known column names are then determined and their visibility is coordinated with the stored preferences. Each column will then be shown or hidden according to its saved setting. This way, the column states are not fleeting, but consistent between sessions and restarts.

This integration is deliberately implemented discreetly. The dialog is invoked only when the user becomes active, and the service automatically takes into account the saved settings. In this way, the application is not overloaded with new interactions, but its behavior is organically expanded by a context-aware memory.

One aspect of this architecture is the way it maintains the balance between control and simplicity. The user does not need to load or save any configuration – everything happens in the background. The view remains responsive, the server records the state, and the dialog acts as an interface between the two. This invisible connection between the surface and the persistence layer gives the application the lightness that keeps it intuitive and efficient despite the growing variety of functions.

Cheers Sven

]]>EclipseStoreServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-detail-dialog-part-2/Sun, 07 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-detail-dialog-part-2/Client contract from a UI perspective In this project, the user interface not only serves as a graphical layer on top of the backend, but is also part of the overall contract between the user, the client, and the server. This part focuses on the data flow from the UI’s perspective: how inputs are translated into structured requests, how the client forwards them, and what feedback the user interface processes.<![CDATA[

Client contract from a UI perspective

In this project, the user interface not only serves as a graphical layer on top of the backend, but is alsopart of the overall contract between the user, the client, and the server. This part focuses on the data flow from the UI’s perspective: how inputs are translated into structured requests, how the client forwards them, and what feedback the user interface processes.

1. Client contract from a UI perspective
2. Client layer (URLShortenerClient): Extensions
3. Server API and Handler
4. Persistence and Store Implementations
   1. InMemory Implementation
   2. EclipseStore Implementation
   3. Uniformity and Compatibility
   4. Advantages of the approach
5. Domain Model and Defaults
   1. ShortUrlMapping as a central link
   2. DefaultValues – Central System Constants
   3. AliasPolicy with Logging
6. JSON serialisation and deserialization
   1. Extending Serialisation in JsonUtils
   2. Extending Deserialization
   3. Purge of incoming JSON data
   4. Optimisation of JSON output
   5. Consistency and interoperability
7. Security and robustness in the UI flow
   1. Input validations in the generation dialogue
   2. Defensive navigation in the detail dialogue
   3. Dealing with the Clipboard API
   4. Consistent feedback and fault tolerance
8. Result

The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-03

Here’s the screenshot of the version we’re implementing now.

The basis of the contract is the class ShortenRequest, which was extended in this development step with the new field expiresAt. This field serves as a central repository for expiration dates and is entirely optional – meaning existing clients will continue to function even without this attribute. The UI client is thus both backwards-compatible and future-proof.

javaCopy

publicclassShortenRequest{ privateStringurl; privateStringshortURL; privateInstantexpiresAt; publicShortenRequest(Stringurl,StringshortURL,InstantexpiresAt){   this.url=url;   this.shortURL=shortURL;   this.expiresAt=expiresAt; } publicInstantgetExpiresAt(){returnexpiresAt;} publicvoidsetExpiresAt(InstantexpiresAt){this.expiresAt=expiresAt;}}

TheCreateView passes this object to the URLShortenerClient, which handles communication with the server. The decisive factor here is that the user interface does not establish HTTP connections directly but delegates this task to a dedicated client component. This keeps the UI lean and testable, while the client is centrally responsible for logging, error handling and request generation. The central interface is the createCustomMapping method, which maps the extended contract:

javaCopy

publicShortUrlMappingcreateCustomMapping(Stringalias,Stringurl,InstantexpiredAt)throwsIOException{ logger().info("Create custom mapping alias='{}' url='{}' expiredAt='{}'",alias,url,expiredAt); URLshortenUrl=serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL(); HttpURLConnectionconnection=(HttpURLConnection)shortenUrl.openConnection(); connection.setRequestMethod("POST"); connection.setDoOutput(true); connection.setRequestProperty(CONTENT_TYPE,JSON_CONTENT_TYPE); varshortenRequest=newShortenRequest(url,alias,expiredAt); Stringbody=shortenRequest.toJson(); try(OutputStreamos=connection.getOutputStream()){   os.write(body.getBytes(UTF_8)); } intstatus=connection.getResponseCode(); if(status==200||status==201){   try(InputStreamis=connection.getInputStream()){     StringjsonResponse=newString(is.readAllBytes(),UTF_8);     ShortUrlMappingshortUrlMapping=fromJson(jsonResponse,ShortUrlMapping.class);     returnshortUrlMapping;   } } if(status==409){   thrownewIllegalArgumentException("Alias already in use"); } thrownewIOException("Unexpected status: "+status);}

This example illustrates how precisely the UI and the client are coordinated. The UI passes a fully populated domain instance (ShortenRequest) that contains all the required fields. The client handles the serialisation, performs the communication, and returns a ShortUrlMapping in response. The UI then displays the relevant data immediately.

A central design principle in this interaction is“data instead of commands”. The UI does not send specific control commands, but always sends complete data objects. The backend decides how to operate based on these objects. This decoupling has several advantages:

1. Extensibility: New fields (e.g. expiresAt) can be added without breaking existing APIs.
2. Traceability: Every operation is fully traceable via the Request object.
3. Security: The client can validate inputs before converting them into HTTP requests.

In the UI, the client’s responses are used to provide feedback and update the current view. Not only is the shortcode displayed, but the entire mapping object is also displayed, which already contains all the values calculated on the server side. The user interface thus always remains in line with the actual system state.

This pattern –a clear contract structure between UI, client and server – creates stability and maintainability in the long term. Changes to the backend API do not require adjustments to the UI logic, as long as the underlying data contract remains unchanged. This establishes a binding communication path that enables technical evolution without impairing user interaction.

Client layer (URLShortenerClient): Extensions

The client layer forms the technical bridge between the user interface and the server. It translates data objects into HTTP requests, monitors the communication, and transfers the results back into domain objects. This chapter shows how the existing class URLShortenerClient has been extended to support the new expiration concept (expiresAt) while ensuring clean, valid communication with the server.

The starting point was the existing function createCustomMapping(String alias, String url), which an overloaded variant has now supplemented. This accepts an additional expiration date (Instant expiredAt) and performs all necessary steps to transfer the data to the server in a complete and compliant manner.

javaCopy

publicShortUrlMappingcreateCustomMapping(Stringalias,Stringurl,InstantexpiredAt)   throwsIOException{ logger().info("Create custom mapping alias='{}' url='{}' expiredAt='{}'",alias,url,expiredAt); if(alias!=null&&!alias.isBlank()){   varvalidate=AliasPolicy.validate(alias);   if(!validate.valid()){     varreason=validate.reason();     thrownewIllegalArgumentException(reason.defaultMessage);   } } URLshortenUrl=serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL(); HttpURLConnectionconnection=(HttpURLConnection)shortenUrl.openConnection(); connection.setRequestMethod("POST"); connection.setDoOutput(true); connection.setRequestProperty(CONTENT_TYPE,JSON_CONTENT_TYPE); varshortenRequest=newShortenRequest(url,alias,expiredAt); Stringbody=shortenRequest.toJson(); logger().info("createCustomMapping - body - '{}'",body); try(OutputStreamos=connection.getOutputStream()){   os.write(body.getBytes(UTF_8)); } intstatus=connection.getResponseCode(); if(status==200||status==201){   try(InputStreamis=connection.getInputStream()){     StringjsonResponse=newString(is.readAllBytes(),UTF_8);     logger().info("createCustomMapping - jsonResponse - {}",jsonResponse);     ShortUrlMappingshortUrlMapping=fromJson(jsonResponse,ShortUrlMapping.class);     logger().info("shortUrlMapping .. {}",shortUrlMapping);     returnshortUrlMapping;   } } if(status==409)   thrownewIllegalArgumentException("Alias already in use"); } thrownewIOException("Unexpected status: "+status);}

With this method, the extended expiration date is seamlessly integrated into the existing communication flow. The UI calls this method via CreateView, keeping the extension transparent to the user – the new functionality is immediately available without changing the user experience.

In addition to improving the method signature, theconsistency of HTTP communication has also been improved. Instead of manually setting headers, the constant JSON_CONTENT_TYPE is now used consistently to avoid format errors and ensure unique typing. This standardisation reduces the risk of inconsistent requests and facilitates later protocol extensions (e.g., for authenticating or signing requests).

Another critical point is logging. The URLShortenerClient logs all relevant steps – from request creation to response processing. This transparency is crucial for understanding the exact process in the event of an error. Especially during the development phase and when integrating new features such as expiresAt, logging provides valuable insights into the timing, format and status of the data transfer.

A typical log snippet might look like this:

sqlCopy

INFOCreatecustommappingalias='test123'url='https://example.com'expiredAt='2025-12-31T23:59:00Z'INFOcreateCustomMapping-body-'{"url": "https://example.com", "alias": "test123", "expiresAt": "2025-12-31T23:59:00Z"}'INFOcreateCustomMapping-jsonResponse-'{"shortCode": "ex-9A7", "originalUrl": "https://example.com", "expiresAt": "2025-12-31T23:59:00Z"}'INFOshortUrlMapping..ShortUrlMapping[shortCode=ex-9A7,url=https://example.com,expiresAt=2025-12-31T23:59:00Z]

This structured logging follows a clear pattern that makes all the steps involved traceable. It is intended not only for debugging, but also for integration into an audit or monitoring solution in the long term.

In conclusion, it should be noted that the client layer is now fullycontext-sensitive. It recognises optionally passed expiration dates, validates inputs, communicates clearly structured JSON payloads, and correctly interprets server-side responses. This creates a robust, well-defined interface between the user interface and the backend that can easily accommodate future extensions, such as custom policies or metadata.

Server API and Handler

The server layer forms the backbone of the entire architecture. It receives requests, interprets the data structures, and coordinates the creation, storage, or deletion of short links. With the introduction of the expiration date (expiresAt), the API has been extended by a significant semantic dimension. The goal was to integrate this new information into the existing request-response flow without causing compatibility issues with older clients.

The central point of contact for incoming POST requests to create a short link is the ShortenHandler. This handles the JSON payload, performs validations, and interacts with the UrlMappingStore. In doing so, the processing has been extended to correctly extract the expiration date from the JSON object and pass it to persistence.

xmlCopy

final String body = readBody(ex.getRequestBody());ShortenRequest req = fromJson(body, ShortenRequest.class);if (isNullOrBlank(req.getUrl())) {  writeJson(ex, BAD_REQUEST, "Missing 'url'");  return;}final Result<ShortUrlMapping> urlMappingResult =    store.createMapping(req.getShortURL(), req.getUrl(), req.getExpiresAt());urlMappingResult  .ifPresentOrElse(success -> logger().info("mapping created success {}", success.toString()),                   failed -> logger().info("mapping created failed - {}", failed));urlMappingResult  .ifSuccess(mapping -> {    final Headers h = ex.getResponseHeaders();    h.add("Location", "/r/" + mapping.shortCode());    writeJson(ex, fromCode(201), toJson(mapping));  })  .ifFailure(errorJson -> {    try {      var parsed = JsonUtils.parseJson(errorJson);      var errorCode = Integer.parseInt(parsed.get("code"));      var message = parsed.get("message");      writeJson(ex, fromCode(errorCode), message);    } catch (Exception e) {      writeJson(ex, CONFLICT, errorJson);    }  });

The ShortenHandler deliberately does not use a framework, but works with native Java APIs (HttpExchange, HttpURLConnection) to keep the control flow completely transparent. This decision is not only for traceability but also enables a precise understanding of how HTTP works at the lowest level. The code clearly shows the steps of the server cycle: read body, deserialise, validate, invoke domain logic, and write response.

Another example is the ListHandler, which has been slightly modified to take advantage of the modern Sequenced API (List.getFirst()), thus increasing readability:

xmlCopy

private static String first(Map<String,List<String>> q, String key) {  var v = q.get(key);  return (v == null || v.isEmpty()) ? null : v.getFirst();}

Special attention is paid to robust JSON data processing. Here, it has been ensured that line breaks and escape sequences do not cause parsing problems. In the JsonUtils module, a cleanup step was introduced before parsing:

s = s.replaceAll("\\n", “”);

This prevents multi-line JSON data from leading to errors – a typical stumbling block for APIs that process manually generated or logged payloads.

Overall, the server API deliberately remainsflat and declarative. Each operation represents a clearly defined domain action; there is no excessive branching or hidden state change. By adding expiresAt, this style is retained, and the system continues to react deterministically: A request creates a mapping, optionally with an expiration date, and returns the complete record as JSON.

This simplicity is not a coincidence, but an expression of a design principle that runs through all levels of the application:explicit data flows instead of implicit magic. The result is a system that remains understandable for both users and developers and can be reliably expanded.

Persistence and Store Implementations

The persistence layer is the foundation on which the entire system’s reliability rests. With the introduction of the expiration date (expiresAt), it had to be expanded accordingly so that this new information can be reliably stored, queried and evaluated – regardless of whether the data is stored in memory or in a persistent database such as EclipseStore.

At the center of this layer is the UrlMappingUpdater interface, which has been extended by a new method. This method adds the Instant expiredAt parameter to the previous signatures, so that the persistence layer can now explicitly handle expiration times.

xmlCopy

public interface UrlMappingUpdater {  Result<ShortUrlMapping> createMapping(String originalUrl);  Result<ShortUrlMapping> createMapping(String alias, String url);  Result<ShortUrlMapping> createMapping(String alias, String url, Instant expiredAt);  boolean delete(String shortCode);}

This clearly states that every implementation must process flow information. This adaptation follows the principle ofcontract-based design – the interface defines which capabilities the specific implementation must possess without prescribing their technical details.

InMemory Implementation

The first customization was done in the InMemoryUrlMappingStore. This class is primarily used for tests and volatile runtime environments and stores all mappings in a ConcurrentHashMap. By extending the createMapping methods, expiration data is now correctly transferred to the MappingCreator.

xmlCopy

@Overridepublic Result<ShortUrlMapping> createMapping(String alias, String originalUrl, Instant expiredAt) {  logger().info("alias: {} - originalUrl: {} - expiredAt: {} ", alias, originalUrl, expiredAt);  return creator.create(alias, originalUrl, expiredAt);}

In the MappingCreator itself, the expiration time is integrated into the ShortUrlMapping and stored directly when created:

xmlCopy

public Result<ShortUrlMapping> create(String alias, String url, Instant expiredAt) {  logger().info("createMapping - alias='{}' / url='{}' / expiredAt='{}'", alias, url, expiredAt);  final String shortCode;  if (!isNullOrBlank(alias)) {    if (repository.containsKey(alias)) {      return Result.failure("Alias already exists");    }    shortCode = alias;  } else {    shortCode = generator.generate();  }  var mapping = new ShortUrlMapping(shortCode, url, Instant.now(clock), Optional.ofNullable(expiredAt));  store.accept(mapping);  return Result.success(mapping);}

This pattern shows that expiration information, if any, is directly part of the ShortUrlMapping domain object. The code is deliberately simple: no additional state, no special treatment, but only an optional value.

EclipseStore Implementation

The EclipseStoreUrlMappingStore has also been adapted for permanent storage. The same principle applies here, but with a focus on long-term persistence.

xmlCopy

@Overridepublic Result<ShortUrlMapping> createMapping(String alias, String originalUrl, Instant expiredAt) {  logger().info("alias: {} - originalUrl: {} - expiredAt: {}", alias, originalUrl, expiredAt);  return creator.create(alias, originalUrl, expiredAt);}

Tightly coupling with the same MappingCreator ensures complete consistency in behaviour between the InMemory and EclipseStore variants. The only difference is the storage duration: While the data in the InMemory store is lost on restart, it remains persistent in the EclipseStore.

Uniformity and Compatibility

A central goal of these adaptations was thecomplete equal treatment of all persistent species. Whether it’s testing, development mode, or production, all paths use the same objects and methods. This eliminates the risk of divergent logic across different storage types. Changes to the domain, such as the introduction of expiresAt, therefore only have to be implemented in one place.

Advantages of the approach

This consistent standardization brings several advantages:

1. Transparency: Every mapping operation is traceable and documented in the log.
2. Consistency: InMemory and EclipseStore stores behave identically.
3. Extensibility: New storage mechanisms (e.g., SQL, key-value store, cloud) can be easily added as long as they fulfil the interface contract.

With this extension, the persistence layer becomes the system’s reliable backbone. The expiration date is now a full-fledged part of the data model – precisely recorded, securely stored and retrievable at any time. This means that future functions such as automatic cleaning of expired entries or time-based statistics can be implemented directly on this basis.

Domain Model and Defaults

The class ShortenRequest has been extended to include the field expiresAt. This field allows you to pass an optional expiration time, which is set by the user in the UI and delivered to the server as an instant in JSON format. This information thus becomes a full-fledged component of the data model and can be further processed at both the transport layer and the persistence layer.

javaCopy

publicclassShortenRequest{ privateStringurl; privateStringshortURL; privateInstantexpiresAt; publicShortenRequest(Stringurl,StringshortURL,InstantexpiresAt){   this.url=url;   this.shortURL=shortURL;   this.expiresAt=expiresAt; } publicInstantgetExpiresAt(){returnexpiresAt;} publicvoidsetExpiresAt(InstantexpiresAt){this.expiresAt=expiresAt;} publicStringtoJson(){   vara=shortURL==null?"\"null\"":"\""+JsonUtils.escape(shortURL)+"\"";   varb=expiresAt==null?"\"null\"":"\""+JsonUtils.escape(expiresAt.toString())+"\"";   return"""        {          \"url\": \"%s\",          \"alias\": %s,          \"expiresAt\": %s        }        """.formatted(JsonUtils.escape(url),a,b); }}

It is important to note that the expiresAt value is not mandatory. This keeps existing clients, and server calls compatible, even if they don’t set the field. The domain model was deliberately designed to remain backwards-compatible and extensible, an essential principle when introducing new features.

ShortUrlMapping as a central link

The ShortUrlMapping class represents the central data element between the client and the server. It contains all the relevant information of a short link: the generated shortcode, the destination URL, the creation date and, optionally, the expiration date. By using Optional, the possible absence of an expiration date is explicitly modelled.

public record ShortUrlMapping(String shortCode, String originalUrl, Instant createdAt, Optional expiresAt) { }

This decision underscores the model’s functional properties: an immutable data element that is fully defined only when it is created. Changes to a mapping are always made through new creations or explicit updates – never through silent mutations.

DefaultValues – Central System Constants

In addition to the model classes, the DefaultValues class has also been extended. It contains constants that are used throughout the application, especially the base URL for generated short links.

javaCopy

publicfinalclassDefaultValues{ TODO-mustbeeditablebyuser publicstaticfinalStringSHORTCODE_BASE_URL="https://3g3.eu/"; publicstaticfinalintADMIN_SERVER_PORT=9090; publicstaticfinalStringADMIN_SERVER_HOST="localhost"; publicstaticfinalStringADMIN_SERVER_PROTOCOL="http"; morepathdefinitions...}

The constant SHORTCODE_BASE_URL serves as the basic component for generating and displaying short links in the user interface. Although it is currently statically defined, it has already been noted that it will be dynamically configurable in a future iteration. This lays the foundation for flexible deployment scenarios in which different environments (e.g., development, test, production) can use their own base URLs.

AliasPolicy with Logging

Another component of the domain model is the AliasPolicy, which defines rules for valid aliases. Logging has been added as part of the enhancements to make the validation processes easier to understand:

javaCopy

publicfinalclassAliasPolicyimplementsHasLogger{ publicstaticValidationvalidate(Stringalias){   HasLogger.staticLogger().info("validate - {}",alias);   if(alias==null||alias.isBlank())returnValidation.fail(Reason.NULL_OR_BLANK);   if(alias.length()<MIN)returnValidation.fail(Reason.TOO_SHORT);   if(alias.length()>MAX)returnValidation.fail(Reason.TOO_LONG);   if(!ALLOWED_PATTERN.matcher(alias).matches())returnValidation.fail(Reason.INVALID_CHARS);   returnValidation.success(); }}

This logging makes faulty aliases immediately visible, making troubleshooting the interaction between the UI and the backend much easier.

With these adjustments, the domain model becomes a robust, clearly structured core of the application. The central entity ShortUrlMapping fully reflects the real-world state of a shortlink, while ShortenRequest controls the creation of new entries and provides system-wide constants to DefaultValues. All extensions remain consistent with the original design principle: simple, functional structures that precisely define what a user can create, modify, or retrieve.

JSON serialisation and deserialization

The reliability of communication between the components of an application depends crucially on the quality of the serialisation layer. This chapter describes how JSON processing has been extended and stabilised to safely transport the new expiresAt field while improving code readability and error tolerance.

Extending Serialisation in JsonUtils

The JsonUtils class forms the backbone of JSON processing in the project. It provides both generic helper methods and specific serialisation routines for the most critical domain objects. With the introduction of the expiration date, it was necessary to ensure this field was correctly integrated into JSON documents without affecting legacy data formats.

In the method for serialising ShortenRequest, the field expiresAt has therefore been added:

javaCopy

if(dtoinstanceofShortenRequestreq){ Map<String,Object>m=newLinkedHashMap<>(); m.put("url",req.getUrl()); m.put("alias",req.getShortURL()); m.put("expiresAt",req.getExpiresAt()); returntoJson(m);}

This change will automatically include the expiration date, if it exists. If it is not set, the field appears as null in the JSON and thus remains syntactically valid. This explicit representation of null values improves readability and allows the server to clearly distinguish between “not set” and “deliberately empty”.

Extending Deserialization

Analogous to serialisation, deserialization has also been extended to read expiresAt from JSON data correctly. In the fromJson method, the customisation is done:

javaCopy

if(type==ShortenRequest.class){ Stringurl=m.get("url"); Stringalias=m.get("alias"); InstantexpiresAt=parseInstantSafe(m.get("expiresAt")); return(T)newShortenRequest(url,alias,expiresAt);}

The parseInstantSafe function converts an ISO-8601 string to an instant object and handles invalid or empty values gracefully. This error resistance is significant for clients that may send different or older JSON structures.

Purge of incoming JSON data

A common problem with APIs is reading in JSON data that contains unintentional line breaks or extra escape characters. To prevent parsing errors, simple preprocessing has been introduced in JsonUtils.parseJson:

s = s.replaceAll("\\n", “”);

This step removes all line breaks before the parser runs. This reliably recognises and correctly interprets both manually formatted JSON files and logged messages. This customisation makes the system more robust against inconsistent formatting that is common in real-world environments. (At this point, however, I am aware that it is far from sufficient…)

Optimisation of JSON output

As part of these changes, the toJsonListing method has also been revised. Instead of a complicated StringBuilder structure, a simple, readable string concatenation is now used:

javaCopy

return"{"+   "\"mode\":\""+escape(mode)+"\","+   "\"count\":"+count+","+   "\"items\":"+toJsonArrayOfObjects(items)+   "}";

This simplification reduces the susceptibility to errors and makes debugging easier. Especially in systems that do not require frameworks, code readability is a decisive factor for maintainability and error diagnosis.

Consistency and interoperability

An essential aspect of the revision of serialisation was maintaininginteroperability across different clients and API versions. Since all fields are still serialised on a string-by-string basis and the JSON structure is explicit, the data exchange remains fully compatible even with older clients. This means that a client that does not send expiresAt will be accepted by the server, and a server that does not expect the field will ignore it.

This loose coupling between transmitter and receiver is a central design goal of the project. It allows incremental expansions without updating all components at once.

The revision to JSON processing strengthens the application’s robustness and future-proofing. By specifically extending JsonUtils to include the expiresAt field, cleaning incoming data, and simplifying the output, serialisation is now both technically stable and semantically precise. It thus meets the requirements of a modern, evolvable interface that remains clearly comprehensible for both automated processes and human readers.

Security and robustness in the UI flow

As the complexity of the user interface grows, so does the responsibility to ensure that all interactions remain predictable, valid, and stable. This chapter explains how security considerations and robustness were built directly into the UI flow – from input validation to defensive navigation decisions to dealing with external browser APIs.

Input validations in the generation dialogue

The most important security aspect in the UI is validating user input. As soon as a new short link is created, the application checks whether the entered URL contains a valid scheme. This prevents potential attacks by manipulated or unsupported protocols at an early stage.

javaCopy

binder.forField(urlField)   .asRequired("URL must not be empty")   .withValidator(url->url.startsWith("http://")||url.startsWith("https://"),                  "Only HTTP(S) URLs allowed")   .bind(ShortenRequest::getUrl,ShortenRequest::setUrl);

This simple validation only accepts URLs that can be reached via secure or well-defined transport protocols. This protects both the user and the application from unwanted interactions with unsafe targets. However, it is not yet full input validation.

Another part of the validation concerns the expiration date. Here, it is ensured that a selected point in time is always in the future:

javaCopy

if(exp.isPresent()&&exp.get().isBefore(Instant.now())){ Notification.show("Expiry must be in the future"); returnfalse;}

This mechanism protects against incorrect entries and inconsistent data states, especially when users edit the form multiple times or select times that have already expired.

Defensive navigation in the detail dialogue

The detail dialogue (DetailsDialog) also follows the principle of secure interaction. If a saved URL is opened via the “Open” button, this is done exclusively by calling the method UI.getCurrent().getPage().open(), but only if the destination is clearly recognised as an HTTP or HTTPS link. This prevents internal or local resources from being accidentally or intentionally called via the UI.

javaCopy

openBtn.addClickListener(_->{ if(originalUrl.startsWith("http://")||originalUrl.startsWith("https://")){   fireEvent(newOpenEvent(this,shortCode,originalUrl));   getUI().ifPresent(ui->ui.getPage().open(originalUrl,"_blank")); }else{   Notification.show("Invalid URL scheme"); }});

This decision strengthens the separation between internal and external resources. User actions are controlled to prevent unwanted side effects outside the application.

Dealing with the Clipboard API

Another security-relevant topic is how to use the nativeClipboard API. For data protection reasons, this is only available in secure browser contexts, i.e. via HTTPS or localhost. The application uses this API to copy shortcodes and URLs to the clipboard conveniently. If access is not allowed in the current context, the call does not result in an error, but is silently discarded – an example ofdefensive programming behaviour in the UI.

UI.getCurrent().getPage().executeJs(“navigator.clipboard.writeText($0)”, SHORTCODE_BASE_URL + m.shortCode());

This non-blocking call avoids JavaScript errors and keeps the UI stable even if the browser rejects the action. The application always responds in a controlled manner and remains in a valid state.

Consistent feedback and fault tolerance

A core element of robustness is thefeedback system. Every user command – whether successful or incorrect – triggers visual feedback. The application consistently uses the Vaadin notification component for this purpose. It provides information about validation errors, successful copying operations and system messages without interrupting the workflow. This asynchronous reporting system supports the idea that a user can continue at any time, even if a single operation fails.

Notification.show(“Alias already assigned or error saving”, 3000, Notification.Position.MIDDLE);

This type of error communication avoids frustration and contributes to perceived stability. The user remains informed, but never blocked.

The measures described in this chapter – input validations, defensive navigation, and safe clipboard use – share a common goal:robustness through caution. Every action in the user interface is checked, every external interface is secured, and every user interaction is clearly reported. This allows the UI to achieve a high level of error tolerance without sacrificing ease of use. This balance of security and user-friendliness concludes the technical implementation of the detailed dialogue. It lays the foundation for the upcoming enhancements, in which security and user experience will continue to go hand in hand.

Result

With today’s Advent calendar, the application has reached a decisive degree of maturity. While the first parts mainly laid the structural and functional foundations, today it is all about the depth of detail and the quality of interaction. The focus was on transitioning from a technically functional interface to a well-thought-out, user-centric application.

The newly introduceddetail dialogue marks a central turning point: it allows viewing individual entries in context without leaving the overview. This concept combines technical clarity with ease of use, creating a modular structure that can easily accommodate future expansions. By using events to decouple the components, the architecture remains clean, traceable and testable.

The integration of the expiration dateis also proving to be a milestone in the system’s functional development. From input in the UI to the client layer to persistence and JSON processing, the new parameter has been consistently integrated into all layers. The original design was deliberately kept simple – each layer knows only its own responsibility, and none contains logic that anticipates another layer. The result is a consistent, precise data flow that combines technical precision with semantic transparency.

Another critical advance is improvingthe user experience (UX). Consistent interaction patterns, precise feedback, and security-conscious behaviour make the application both intuitive and trustworthy. The combination of immediate feedback, non-blocking error handling, and a harmonious visual-functional design shows how technological rigour and user focus can go hand in hand.

From an architectural point of view, this part of the Advent calendar illustrates that even in small projects**, cleanliness, coherence, and expandability** are the key success factors. The clear separation between UI, client, server, and persistence not only enables efficient maintenance but also opens the way for future modules – such as administrative views, bulk operations, or security policies for user groups.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-detail-dialog-part-1/Sat, 06 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-detail-dialog-part-1/Classification and objectives from a UI perspective Today’s Advent Calendar Day focuses specifically on the interaction level prepared in the previous parts. While the basic structure of the user interface and the layout were defined at the beginning, and the interactive table view with sorting, filtering and dynamic actions was subsequently established, it is now a matter of making the transition from overview to detailed observation consistent. The user should no longer only see a tabular collection of data points, but should receive a view tailored to the respective object that enables contextual actions.<![CDATA[

Classification and objectives from a UI perspective

Today’s Advent Calendar Day focuses specifically on the interaction level prepared in the previous parts. While the basic structure of the user interface and the layout were defined at the beginning, and the interactive table view with sorting, filtering and dynamic actions was subsequently established, it is now a matter of making the transition from overview to detailed observation consistent. The user should no longer only see a tabular collection of data points, but should receive a view tailored to the respective object that enables contextual actions.

1. Classification and objectives from a UI perspective
2. OverviewView: Interactive enhancements
3. Detail view as a standalone UI component (DetailsDialog)
4. CreateView: Expiration date in the UI
5. Navigation and package structure
6. Interaction patterns and UX coherence
   1. Uniform copying behaviour
   2. Context menus and multiple interactions
   3. Consistency of feedback
   4. HTTPS Requirement for Clipboard

The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-03

Here’s the screenshot of the version we’re implementing now.

The central goal of this chapter is to improve the user experience without sacrificing the simplicity of what has been done so far. The interface is extended with adetailed view implemented as astandalone dialogue , thereby deliberately creating a clear framework between the overall list and the individual object. This decision follows the principle of the cognitive separation of information spaces: an overview serves to orient and find relevant entries. At the same time, the detailed view creates an isolated, focused working environment.

From an architectural point of view, this extension is an essential step towards acomponent-oriented UI in which each functional unit – such as creating, displaying or deleting short links – is represented by its own, clearly defined components. The new dialogue includes all the necessary UI elements, validations, and event flows for displaying and interacting with a single ShortUrlMapping object. This decoupling not only enables better testability and reusability but also reduces the cognitive overhead in the primary grid, which previously had to record all interactions directly.

The dialogue serves as aninteractive interface between the visual representation and the underlying data model. It reflects the current object, provides copy and navigation actions, and provides visual feedback on the entry’s status – for example, via a colour-coded expiration date. This modular structure allows the user to check details, perform actions, or remove erroneous entries without losing the application’s context.

OverviewView: Interactive enhancements

With the third expansion stage of the user interface, the previous list view is not only expanded, but also functionally upgraded. TheOverviewView forms the foundation of the daily work with the created short links and is supplemented in this step by several mechanisms that make the behaviour of the application more natural and efficient.

A central aspect concernsthe reactivity of the search fields. In previous versions, search queries were triggered immediately after each input, which was technically correct but inefficient in practice, using the ValueChangeMode.LAZY, in combination with a 400-millisecond delay, the system does not react until the user has completed their input. This delay serves as a natural “pause for thought” and prevents unnecessary updates to the grid. In addition, theenter-key flow has been activated so that a targeted search confirmation can be performed via the keyboard – a typical usage pattern in the professional environment.

javaCopy

codePart.setValueChangeMode(ValueChangeMode.LAZY);codePart.setValueChangeTimeout(400);codePart.addValueChangeListener(e->refresh());urlPart.setValueChangeMode(ValueChangeMode.LAZY);urlPart.setValueChangeTimeout(400);urlPart.addValueChangeListener(e->refresh());

A second improvement concerns thevariety of interactions in the grid. Instead of acting exclusively via buttons, the user can now open a data record bydouble-clicking or pressingEnter. This form of interaction is not only faster, but also meets the expectations you are used to from desktop applications. To further simplify operation, acontext menu has been added that automatically offers the appropriate actions when right-clicking: View details, open target URL, copy shortcode, or delete the entry.

xmlCopy

grid.addItemDoubleClickListener(ev -> openDetailsDialog(ev.getItem()));grid.addItemClickListener(ev -> { if (ev.getClickCount() == 2) openDetailsDialog(ev.getItem());});GridContextMenu<ShortUrlMapping> menu = new GridContextMenu<>(grid);menu.addItem("Show details", e -> e.getItem().ifPresent(this::openDetailsDialog));menu.addItem("Open URL", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().open(m.originalUrl(), "_blank")));menu.addItem("Copy shortcode", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", m.shortCode())));menu.addItem("Delete...", e -> e.getItem().ifPresent(m -> confirmDelete(m.shortCode())));

In addition to functionality, the grid column layout has been revised. The shortcode is now displayed in amonospace font , which makes it easier to quickly grasp and compare visually. In addition, a small copy symbol allows copying the entire short link to the clipboard with a mouse click. JavaScript is used to call up the browser’s native clipboard service.

javaCopy

grid.addComponentColumn(m->{varcode=newSpan(m.shortCode());code.getStyle().set("font-family","ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace");varcopy=newButton(newIcon(VaadinIcon.COPY));copy.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE);copy.getElement().setProperty("title","Copy ShortUrl");copy.addClickListener(_->{UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)",SHORTCODE_BASE_URL+m.shortCode());Notification.show("Shortcode copied");});varwrap=newHorizontalLayout(code,copy);wrap.setSpacing(true);wrap.setPadding(false);returnwrap;}).setHeader("Shortcode").setAutoWidth(true).setFrozen(true).setResizable(true).setFlexGrow(0);

In the column with the original URL, the layout has been changed to anelliptical display : Long URLs are truncated, but remain fully visible via the tooltip. This detail improves readability without losing information.

javaCopy

grid.addComponentColumn(m->{vara=newAnchor(m.originalUrl(),m.originalUrl());a.setTarget("_blank");a.getStyle().set("white-space","nowrap").set("overflow","hidden").set("text-overflow","ellipsis").set("display","inline-block").set("max-width","100%");a.getElement().setProperty("title",m.originalUrl());returna;}).setHeader("URL").setFlexGrow(1).setResizable(true);

Particularly noteworthy is the newExpires column , whichcolour-coded status indicators have visually supplemented. These are based on Lumo badges and show the remaining time until a link expires: green for active, yellow for expiring soon, and red for expired.

javaCopy

grid.addComponentColumn(m->{varpill=newSpan(m.expiresAt().map(ts->{vardays=Duration.between(Instant.now(),ts).toDays();if(days<0)return"Expired";if(days==0)return"Today";return"in "+days+" days";}).orElse("No expiry"));pill.getElement().getThemeList().add("badge pill small");m.expiresAt().ifPresent(ts->{longd=Duration.between(Instant.now(),ts).toDays();if(d<0)pill.getElement().getThemeList().add("error");elseif(d<=3)pill.getElement().getThemeList().add("warning");elsepill.getElement().getThemeList().add("success");});returnpill;}).setHeader("Expires").setAutoWidth(true).setResizable(true).setFlexGrow(0);

Finally, the display’sergonomics have been improved. The grid now works with reduced vertical spacing (compact mode) while retaining its readability thanks to alternating line colours. The line height has been adjusted so that as many entries as possible remain visible even on smaller monitors without making the UI look overloaded.

With these changes, the OverviewView is transformed from a purely management view into a central control tool that provides both a quick overview and deep interaction. This lays the foundation for integrating the detail dialogue – it complements this view with an object-related perspective, while the OverviewView continues to serve as the starting point for navigation.

Detail view as a standalone UI component (DetailsDialog)

Now that the overview list has been expanded with interactive functions, the next logical step is to introduce a standalone UI component focused on the display and management of individual datasets. The goal is to create a modular and reusable view that operates independently of the main view but communicates with it via clearly defined events.

The dialogue is based on Vaadin’s Dialogue class and opens for each selected ShortUrlMapping object. In doing so, it reads out all relevant properties of the passed object – shortcode, target URL, creation time and, optionally, the expiration date. These values are presented in a clearly structured format, supplemented by action buttons to open, copy and delete the entry.

The following excerpt shows the basic structure of the dialogue:

xmlCopy

public class DetailsDialog extends Dialog implements HasLogger { public static final ZoneId ZONE = ZoneId.systemDefault(); private static final DateTimeFormatter DATE_TIME_FMT = DateTimeFormatter.ofPattern("dd.MM.yyyy HH:mm").withZone(ZONE); private final String shortCode; private final String originalUrl; private final Instant createdAt; private final Optional<Instant> expiresAt; private final TextField tfShort = new TextField("Shortcode"); private final TextField tfUrl = new TextField("Original URL"); private final TextField tfCreated = new TextField("Created on"); private final TextField tfExpires = new TextField("Expires"); private final Span statusPill = new Span(); private final Button openBtn = new Button("Open", new Icon(VaadinIcon.EXTERNAL_LINK)); private final Button copyShortBtn = new Button("Copy ShortURL", new Icon(VaadinIcon.COPY)); private final Button copyUrlBtn = new Button("Copy URL", new Icon(VaadinIcon.COPY)); private final Button deleteBtn = new Button("Delete...", new Icon(VaadinIcon.TRASH)); private final Button closeBtn = new Button("Close"); public DetailsDialog(ShortUrlMapping mapping) { Objects.requireNonNull(mapping, "mapping"); this.shortCode = mapping.shortCode(); this.originalUrl = mapping.originalUrl(); this.createdAt = mapping.createdAt(); this.expiresAt = mapping.expiresAt(); setHeaderTitle("Details: " + shortCode); setModal(true); setDraggable(true); setResizable(true); setWidth("720px"); openBtn.addThemeVariants(ButtonVariant.LUMO_PRIMARY); deleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR, ButtonVariant.LUMO_TERTIARY); var headerActions = new HorizontalLayout(openBtn, copyShortBtn, copyUrlBtn, deleteBtn); getHeader().add(headerActions); configureFields(); var form = new FormLayout(); form.add(tfShort, tfUrl, tfCreated, tfExpires, statusPill); form.setColspan(tfUrl, 2); add(form); closeBtn.addClickListener(e -> close()); getFooter().add(closeBtn); wireActions(); }

It is already clear here that the dialogue has a high degree of independence. It initialises its contents directly from the passed data object and uses aFormLayout to structure the fields. The input fields areread-only by default because the dialogue is used primarily for display.

The visual feedback on the expiration status is provided by a so-calledstatus pill , which indicates in colour and text whether a short link is still active, about to expire or has already expired. This is done via a small helper method that provides a status description including a colour scheme:

javaCopy

privatestatuscomputeStatusText(){returnexpiresAt.map(ts->{longd=Duration.between(Instant.now(),ts).toDays();if(d<0)returnnewStatus("Expired","error");if(d==0)returnnewStatus("Expires today","warning");if(d<=3)returnnewStatus("Expires in "+d+" days","warning");returnnewStatus("Valid("+d+" days left)","success");}).orElse(newStatus("No expiry","contrast"));}

This status calculation complements the visual feedback from the summary table and provides a consistent representation across the entire application.

The real added value of the DetailsDialog lies in itsevent-orientation. Instead of the calling view (OverviewView) controlling all actions itself, the actions are defined asVaadin events. This allows the dialogue to send signals such asOpenEvent ,CopyShortcodeEvent ,CopyUrlEvent orDeleteEvent to its environment without knowing what they mean:

xmlCopy

public static class DeleteEvent extends ComponentEvent<DetailsDialog> { public final String shortCode; public DeleteEvent(DetailsDialog src, String sc) { super(src); this.shortCode = sc; }}

In the OverviewView , these events are received and processed:

javaCopy

vardlg=newDetailsDialog(item);dlg.addDeleteListener(ev->confirmDelete(ev.shortCode));dlg.addOpenListener(ev->logger().info("Open URL {}",ev.originalUrl));dlg.addCopyShortListener(ev->logger().info("Copied shortcode {}",ev.shortCode));dlg.addCopyUrlListener(ev->logger().info("Copied URL {}",ev.url));dlg.open();

This creates an apparent decoupling between the presentation and application logic. Dialogue handles representation and interaction; the surrounding view determines how to respond to events.

In summary, theDetailsDialog establishes an architectural principle that goes beyond the specific use case. The combination of a modular UI component, declarative events and a clearly defined data model not only makes the application more flexible, but also more maintainable in the long term. The user benefits from a coherent, clearly structured, detailed view, which makes working with individual entries much more convenient and transparent.

CreateView: Expiration date in the UI

With this step, the creation of new short links receives an important semantic addition: the optionalexpiration date. The aim is to precisely define the intended service life at the time of creation and to transport this information end-to-end – from the UI to the client to the server and into the persistence.

From a UI point of view, the existing CreateView is extended with DatePicker and TimePicker, flanked by a checkbox labelled “No expiry”. This combination allows both the explicit determination of an end date and the conscious decision to set an unlimited validity. A small but crucial UX rule: the time will remain disabled until a date is selected, and both fields will be disabled if “No expiry” is enabled.

Fields and Basic Configuration

xmlCopy

private final TextField urlField = new TextField("Target URL");private final TextField aliasField = new TextField("Alias (optional)");private final Button shortenButton = new Button("Shorten");private final DatePicker expiresDate = new DatePicker("Expires (date)");private final TimePicker expiresTime = new TimePicker("Expires (time)");private final Checkbox noExpiry = new Checkbox("No expiry");private final FormLayout form = new FormLayout();public CreateView() { setSpacing(true); setPadding(true); urlField.setWidthFull(); aliasField.setWidth("300px"); shortenButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY); form.add(urlField, aliasField); configureExpiryFields(); form.setResponsiveSteps( new FormLayout.ResponsiveStep("0", 1), new FormLayout.ResponsiveStep("600px", 2) ); form.setColspan(urlField, 2); var actions = new HorizontalLayout(shortenButton); actions.setAlignItems(Alignment.END); Binder for validations Binder<ShortenRequest> binder = new Binder<>(ShortenRequest.class); ShortenRequest request = new ShortenRequest(); binder.forField(urlField) .asRequired("URL must not be empty") .withValidator(url -> url.startsWith("http://") || url.startsWith("https://"), "Only HTTP(S) URLs allowed") .bind(ShortenRequest::getUrl, ShortenRequest::setUrl); binder.forField(aliasField) .withValidator(a -> a == null || a.isBlank() || a.length()<= AliasPolicy.MAX, "Alias is too long (max " + AliasPolicy.MAX + ")") .withValidator(a -> a == null || a.isBlank() || a.matches(REGEX_ALLOWED), "Only [A-Za-z0-9_-] allowed") .bind(ShortenRequest::getShortURL, ShortenRequest::setShortURL); shortenButton.addClickListener(_ -> { var validated = binder.validate(); if (validated.hasErrors()) return; if (!validateExpiryInFuture()) return; if (binder.writeBeanIfValid(request)) { computeExpiresAt().ifPresent(request::setExpiresAt); var code = createShortCode(request, computeExpiresAt()); code.ifPresentOrElse(c -> { Notification.show("Short link created: " + c); clearForm(binder); }, () -> Notification.show("Alias already assigned or error saving", 3000, Notification.Position.MIDDLE)); } }); add(new H2("Create new short link"), form, actions);}

Encapsulating process logic in small auxiliary methods improves readability and testability. The calculation uses the local time zone and provides an instant that can be processed unchanged on the server side.

xmlCopy

private static final ZoneId ZONE = ZoneId.systemDefault();private void configureExpiryFields() { expiresDate.setClearButtonVisible(true); expiresDate.setPlaceholder("dd.MM.yyyy"); expiresTime.setStep(Duration.ofMinutes(1)); expiresTime.setPlaceholder("HH:mm"); Activate time only when a date is set expiresTime.setEnabled(false); expiresDate.addValueChangeListener(ev -> { boolean hasDate = ev.getValue() != null; expiresTime.setEnabled(hasDate&& !noExpiry.getValue()); }); noExpiry.addValueChangeListener(ev -> { boolean disabled = ev.getValue(); expiresDate.setEnabled(!disabled); expiresTime.setEnabled(!disabled&& expiresDate.getValue() != null); }); form.add(noExpiry, expiresDate, expiresTime);}private Optional<Instant> computeExpiresAt() { if (Boolean.TRUE.equals(noExpiry.getValue())) return Optional.empty(); LocalDate d = expiresDate.getValue(); LocalTime t = expiresTime.getValue(); if (d == null || t == null) return Optional.empty(); return Optional.of(ZonedDateTime.of(d, t, ZONE).toInstant());}private boolean validateExpiryInFuture() { var exp = computeExpiresAt(); if (exp.isPresent()&& exp.get().isBefore(Instant.now())) { Notification.show("Expiry must be in the future"); return false; } return true;}private void clearForm(Binder<ShortenRequest> binder) { urlField.clear(); aliasField.clear(); noExpiry.clear(); expiresDate.clear(); expiresTime.clear(); binder.setBean(new ShortenRequest()); urlField.setInvalid(false); aliasField.setInvalid(false);}

It is important to pass it on transparentlyto the client. Instead of storing the time in the UI, it is transmitted to the server in the request. To do this, the CreateView calls the client with the extended signature. The client validates only if an alias is set and passes the data to the server unchanged.

xmlCopy

private Optional<String> createShortCode(ShortenRequest req, Optional<Instant> expiresAt) { logger().info("createShortCode with ShortenRequest '{}'", req); try { var customMapping = urlShortenerClient.createCustomMapping(req.getShortURL(), req.getUrl(), expiresAt.orElse(null)); return Optional.ofNullable(customMapping.shortCode()); } catch (IllegalArgumentException | IOException e) { logger().error("Error saving", e); return Optional.empty(); }}

On theclient side , the payload is serialised as a ShortenRequest and sent to the /shorten endpoint. The response is not limited to the shortcode; it is also parsed as a full ShortUrlMapping. As a result, the UI immediately knows the server-side confirmed state – including the expiresAt.

javaCopy

publicShortUrlMappingcreateCustomMapping(Stringalias,Stringurl,InstantexpiredAt)throwsIOException{logger().info("Create custom mapping alias='{}' url='{}' expiredAt='{}'",alias,url,expiredAt);if(alias!=null&&!alias.isBlank()){varvalidate=AliasPolicy.validate(alias);if(!validate.valid()){varreason=validate.reason();thrownewIllegalArgumentException(reason.defaultMessage);}}URLshortenUrl=serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL();HttpURLConnectionconnection=(HttpURLConnection)shortenUrl.openConnection();connection.setRequestMethod("POST");connection.setDoOutput(true);connection.setRequestProperty(CONTENT_TYPE,JSON_CONTENT_TYPE);varshortenRequest=newShortenRequest(url,alias,expiredAt);Stringbody=shortenRequest.toJson();try(OutputStreamos=connection.getOutputStream()){os.write(body.getBytes(UTF_8));}intstatus=connection.getResponseCode();if(status==200||status==201){try(InputStreamis=connection.getInputStream()){StringjsonResponse=newString(is.readAllBytes(),UTF_8);ShortUrlMappingshortUrlMapping=fromJson(jsonResponse,ShortUrlMapping.class);returnshortUrlMapping;}}if(status==409){thrownewIllegalArgumentException("Alias already in use");}thrownewIOException("Unexpected status: "+status);}

On theserver side , the ShortenHandler receives the extended request, validates the required fields, and then assigns the store to create it. The response contains the complete mapping object that the client and UI can use immediately.

xmlCopy

final String body = readBody(ex.getRequestBody());ShortenRequest req = fromJson(body, ShortenRequest.class);if (isNullOrBlank(req.getUrl())) { writeJson(ex, BAD_REQUEST, "Missing 'url'"); return;}final Result<ShortUrlMapping> urlMappingResult = store.createMapping(req.getShortURL(), req.getUrl(), req.getExpiresAt());urlMappingResult .ifPresentOrElse(success -> logger().info("mapping created success {}", success.toString()), failed -> logger().info("mapping created failed - {}", failed));urlMappingResult .ifSuccess(mapping -> { final Headers h = ex.getResponseHeaders(); h.add("Location", "/r/" + mapping.shortCode()); writeJson(ex, fromCode(201), toJson(mapping)); }) .ifFailure(errorJson -> { try { var parsed = JsonUtils.parseJson(errorJson); var errorCode = Integer.parseInt(parsed.get("code")); var message = parsed.get("message"); writeJson(ex, fromCode(errorCode), message); } catch (Exception e) { writeJson(ex, CONFLICT, errorJson); } });

In summary**, a consistent, end-to-end process** is created: The user optionally defines an expiration date when making them, the UI validates basic rules, the client transfers the semantics losslessly, and the server persists them reliably. The OverviewView and the DetailsDialog can then display and interpret this information immediately. This adds a central property to the domain without complicating the existing operating flow.

Navigation and package structure

The previous user interface has become much more complex in terms of content due to the detailed dialogue and the extended form functions. To reflect this development, the package structure in the UI module was clearlyreorganised. The goal was not only a logical grouping according to responsibilities, but also a long-term basis for extended navigation concepts and modular extensions.

Previously, the OverviewView was still in the general package com.svenruppert.urlshortener.ui.vaadin.views. With the introduction of detailed dialogue and the growing importance of this area in terms of content, it was included in a separate subpackage, “views.overview “, and moved there. This decision not only creates room for additional components (e.g. context menus, helper dialogues or filters), but also follows the principle offunctional coherence : All classes that together form the overview function are now centrally bundled.

In the code, this step is reflected in the customisation of the import within the MainLayout :

old:

import com.svenruppert.urlshortener.ui.vaadin.views.OverviewView;

new:

import com.svenruppert.urlshortener.ui.vaadin.views.overview.OverviewView;

This small but significant step marks the transition from a purely page-based structure to a component-oriented structure. The MainLayout continues to serve as the central navigation element of the application; the individual views are now more decoupled and can be further developed independently. This creates a clear separation betweenlayout logic (central navigation, menus, visual frameworks) andfunctional logic (display, interaction, data flow).

The route relationships are deliberately kept simple. The OverviewView is still registered under the path /overview and uses the MainLayout as its parent layout element:

javaCopy

@PageTitleOverview@Route(value=OverviewView.PATH,layout=MainLayout.class)publicclassOverviewViewextendsVerticalLayoutimplementsHasLogger{publicstaticfinalStringPATH="overview";// ...}

This configuration keeps navigation consistent with the other parts of the application, such as CreateView, AboutView, or YoutubeView. New views can be easily added without affecting the central navigation mechanism. This ensures maintainability and scalability – two central requirements for an application that grows gradually within the Advent calendar framework.

The MainLayout itself is largely unchanged, but has been updated during the package migration to reflect new imports and menu entries. The referencing of the OverviewView is particularly important, as it represents the entry point of user interaction:

sqlCopy

SideNavItemoverview=newSideNavItem("Overview",OverviewView.class,VaadinIcon.LIST.create());SideNavItemcreate=newSideNavItem("Create",CreateView.class,VaadinIcon.PLUS.create());SideNavItemabout=newSideNavItem("About",AboutView.class,VaadinIcon.INFO_CIRCLE.create());SideNavItemyoutube=newSideNavItem("Youtube",YoutubeView.class,VaadinIcon.YOUTUBE.create());SideNavnav=newSideNav(overview,create,about,youtube);addToDrawer(nav);

This clean separation between routing, structure and presentation lays the foundation for the further development of the user interface. Future features – such as detailed filters, user preferences or administrative functions – can be easily integrated into their own subpackages and namespaces without affecting core navigation. This is the step towards a sustainable UI architecture that specifically supports both growing complexity and extensibility.

Interaction patterns and UX coherence

As the application’s feature density increases, the importance of a consistent user experience grows. While the first days of the Advent calendar laid the technical foundation, so far, the focus has been on functionality. In this chapter, the focus shifts to thecoherence of interaction patterns , i.e. how users interact with the application in a consistent, predictable rhythm.

Central to this is the goal of establishing uniform behaviour for recurring actions. Whether in the overview, in the detail dialogue or in forms – copying, opening or deleting should always feel the same. The application thus conveys reliability, which is particularly crucial for technical users of web-based tools.

Uniform copying behaviour

A good example iscopying URLs and short links. The exact mechanisms are used in both the grid and the detail dialogue: a button with the VaadinIcon.COPY symbol: an asynchronous clipboard action via JavaScript and a discreet confirmation message. This avoids users having to learn different forms of interaction in various views.

javaCopy

copy.addClickListener(_->{UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)",SHORTCODE_BASE_URL+m.shortCode());Notification.show("Shortcode copied");});

Such a detail may seem inconspicuous, but it has a significant impact on the perceived professionalism of the application. The feedback system vianotifications plays a central role here: The user receives an immediate, unobtrusive signal about the success of his action – a kind of visual receipt that creates trust.

Context menus and multiple interactions

Another element of UX coherence is thecontext menu in the summary table. It allows access to the same actions, which can also be accessed via buttons or double-clicks. This redundant but deliberate multiple interaction follows the principle ofuser freedom : Users can choose whether to act via direct icons, the keyboard or the context menu.

xmlCopy

GridContextMenu<ShortUrlMapping> menu = new GridContextMenu<>(grid);menu.addItem("Show details", e -> e.getItem().ifPresent(this::openDetailsDialog));menu.addItem("Open URL", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().open(m.originalUrl(), "_blank")));menu.addItem("Copy shortcode", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", m.shortCode())));menu.addItem("Delete...", e -> e.getItem().ifPresent(m -> confirmDelete(m.shortCode())));

The decision to use context menus is not only aesthetic but also ergonomic: it reduces the grid’s optical density without sacrificing functionality. Actions only appear when they are needed – an approach that is essential in complex management interfaces.

Consistency of feedback

A uniform pattern is also followed for error messages and validations. The system does not abruptly abort user actions; instead, it clearly communicates why an input was not accepted. Example: The expiration date cannot be in the past. This rule is conveyed both visually and by message.

javaCopy

if(exp.isPresent()&&exp.get().isBefore(Instant.now())){Notification.show("Expiry must be in the future");returnfalse;}

Accurate feedback prevents the user from perceiving the system as unpredictable. Every validation, every hint, and every success message follows the same communicative style—short, clear, and polite.

HTTPS Requirement for Clipboard

A technical but essential detail is that theClipboard API only works in modern browsers within secure contexts (HTTPS or localhost). Therefore, the copy feature is deliberately designed to fail elegantly when clipboard access is unavailable. The application does not crash, but responds silently – an aspect that underlines the robustness and professionalism of the user experience.

The mechanisms described in this chapter – consistent feedback, redundant operating options and secure fallbacks – together contribute to a coherent user experience. They form the basis of trust and predictability, two characteristics that are essential as software becomes more complex. Overall, the result is an interface that can be operated intuitively, even as its technical depth in the background continues to increase.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-persistence-part-02/Fri, 05 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-persistence-part-02/Today, we will finally integrate the StoreIndicator into the UI. Vaadin integration: live status of the store Implementation of the StoreIndicator Refactoring inside – The MappingCreator as a central logic. EclipseStore – The Persistent Foundation Additional improvements in the core Before & After – Impact on the developer experience The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-02<![CDATA[

Today, we will finally integrate the StoreIndicator into the UI.

1. Vaadin integration: live status of the store
2. Implementation of the StoreIndicator
3. Refactoring inside – The MappingCreator as a central logic.
4. EclipseStore – The Persistent Foundation
5. Additional improvements in the core
6. Before & After – Impact on the developer experience

The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-02

Here’s the screenshot of the version we’re implementing now.

Vaadin integration: live status of the store

Now that communication between the client and server has been established via the AdminClient, the focus is on integrating with the Vaadin interface. The goal is to make the current system state visible to the user without requiring manual updates. The mechanism is based on cyclic querying, event control, and a reactive UI concept, all fully implemented in Vaadin.

A central component of this integration is the StoreIndicator component. It serves as a visual interface between the user and the system state. Its design combines UI representation, data retrieval, and state management. The structure is implemented within the Vaadin component model, with the indicator implemented as an independent class, “StoreIndicator”. This class extends HorizontalLayout and adds several sub-elements to the layout—an icon, a text label, and optional status information. Their behaviour is closely linked to Vaadin’s lifecycle control.

The central entry point is the onAttach method, which is executed whenever the component is added to the UI tree. At this moment, the polling mechanism starts, calling the AdminClient every 10 seconds to get the current status from the server. Communication remains fully controlled on the server side, while the UI is automatically synchronised.

javaCopy

@OverrideprotectedvoidonAttach(AttachEventattachEvent){refreshOnce();UIui=attachEvent.getUI();ui.setPollInterval(10000);ui.addPollListener(e->refreshOnce());}

Analogous to the onAttach method, onDetach is also an essential part of the StoreIndicator’s lifecycle. This method is executed when the component is removed from the UI tree, such as when switching between views or closing the session. Here, the polling mechanism is cleanly terminated, and resources such as event subscriptions or listeners are released. This prevents background processes from continuing to send data to a defunct UI component. The code illustrates this process:

javaCopy

@OverrideprotectedvoidonDetach(DetachEventdetachEvent){super.onDetach(detachEvent);if(subscription!=null){try{subscription.close();}catch(Exceptionignored){}subscription=null;}}

This implementation ensures that no polling tasks or event listeners remain active after the component is removed. This is especially crucial in server-side frameworks like Vaadin to avoid leaks and unnecessary thread activity. The StoreIndicator thus acts like a well-behaved UI citizen, completing its tasks properly when leaving the scene.

Each polling cycle triggers the refreshOnce method, which uses the AdminClient to determine the current memory state. This checks whether the server returns the mode “EclipseStore” or “InMemory”. The colour scheme is adjusted accordingly, and the symbol is highlighted. The design is based on Vaadin’s Lumo theme system, which enables clean integration with the application’s corporate design via CSS variables.

Another central element is the internal event system, implemented via the StoreEvents and StoreConnectionChanged classes. It allows for loose coupling between components, so that any change to the memory state can be published globally and consumed by other UI elements. The StoreIndicator uses this system to propagate state changes, while the OverviewView, for example, acts as a subscriber and automatically updates itself when the memory state changes.

**StoreEvents.publish(new StoreConnectionChanged(newMode, info.mappings()));

This architecture avoids direct dependencies between UI components. Instead of explicit references or state propagation via view constructors, an event model is used that reduces complexity while increasing the application’s responsiveness. The result is a highly modular, reactive UI structure that reflects data changes on the backend.

The way in which Vaadin is used here as a reactive framework deserves special attention. While classic web frameworks propagate state changes via HTTP requests and complete page reloads, Vaadin uses server-side push to deliver targeted UI updates. In combination with the polling model of the StoreIndicator, a hybrid solution is created: On the one hand, the architecture remains comprehensible and straightforward, and on the other hand, it gives the impression of a live application that reacts to changes in real time.

This integration of event model, UI lifecycle, and data transfer is an example of modern component-oriented design in the Java ecosystem. It shows that reactive behaviour patterns can also be implemented without external reactive frameworks such as Vert.x or Reactor.

Implementation of the StoreIndicator

The indicator’s graphical structure is already fully defined in the constructor. It extends HorizontalLayout and uses Vaadin components such as Icon and Span to create a compact, semantically transparent status display. The structure follows the typical Vaadin composition logic: The graphic elements are created, styled and then inserted into the layout via the add method. Particular attention is paid to legibility and visual consistency:

javaCopy

publicStoreIndicator(AdminClientadminClient){this.adminClient=adminClient;setAlignItems(FlexComponent.Alignment.CENTER);setSpacing(true);setPadding(false);dbIcon.setSize("16px");dbIcon.getStyle().set("color","var(--lumo-secondary-text-color)");badge.getStyle().set("font-size","12px").set("font-weight","600").set("padding","0.2rem 0.5rem").set("border-radius","0.4rem").set("background-color","var(--lumo-contrast-10pct)").set("color","var(--lumo-body-text-color)");details.getStyle().set("font-size","12px").set("opacity","0.8");add(dbIcon,badge,details);}

The actual logic of the status update is contained in the refreshOnce method. It calls the AdminClient to determine the current server state. This returns a StoreInfo object containing the mode and the number of saved mappings. The mode dynamically adjusts the indicator’s colour scheme– green for persistent, blue for volatile, and red for erroneous states. This colour coding is based on Vaadin’s Lumo theming and provides immediate visual feedback.

javaCopy

publicvoidrefreshOnce(){getUI().ifPresent(ui->ui.access(()->{try{StoreInfoinfo=adminClient.getStoreInfo();booleanpersistent="EclipseStore".equalsIgnoreCase(info.mode());badge.setText(persistent?"EclipseStore":"InMemory");if(persistent){badge.getStyle().set("background-color","var(--lumo-success-color-10pct)").set("color","var(--lumo-success-text-color)");dbIcon.getStyle().set("color","var(--lumo-success-color)");}else{badge.getStyle().set("background-color","var(--lumo-primary-color-10pct)").set("color","var(--lumo-primary-text-color)");dbIcon.getStyle().set("color","var(--lumo-primary-color)");}details.setText("· "+info.mappings()+" items");getElement().setAttribute("title",persistent?"Persistent via EclipseStore":"Volatile (InMemory)");varnewMode=persistent?StoreMode.ECLIPSE_STORE:StoreMode.IN_MEMORY;if(newMode!=lastMode){lastMode=newMode;StoreEvents.publish(newStoreConnectionChanged(newMode,info.mappings()));}}catch(Exceptione){badge.setText("Unavailable");badge.getStyle().set("background-color","var(--lumo-error-color-10pct)").set("color","var(--lumo-error-text-color)");dbIcon.getStyle().set("color","var(--lumo-error-color)");details.setText("");getElement().setAttribute("title","StoreInfo endpoint unavailable");if(lastMode!=StoreMode.UNAVAILABLE){lastMode=StoreMode.UNAVAILABLE;StoreEvents.publish(newStoreConnectionChanged(StoreMode.UNAVAILABLE,0));}}}));}

This method is an example of reactive UI design without external frameworks. The combination of ui.access() and PollListener allows periodic updates without blocking the main UI thread model. Error cases are elegantly intercepted via the catch block and propagated via the event bus (StoreEvents.publish), so that other components can also react to failures.

Refactoring inside – The MappingCreator as a central logic.

With the introduction of EclipseStore and the parallel continued existence of the InMemory store, the need arose to standardise the generation logic for short URLs. Earlier versions of the application included this logic multiple times – both in the InMemory store and in the later EclipseStore implementation. This redundancy not only led to maintenance costs but also made uniform error handling more difficult. The solution to this problem was to introduce a dedicated component, theMappingCreator, which acts as a central element between validation, alias policy, and persistence mediation.

The MappingCreator covers the entire process chain from the alias check to the generation of a new short code to the transfer to persistent storage. This component follows the principle of functional composition and is designed to operate independently of the specific storage technology. This means that both the InMemory and EclipseStore approaches can use the exact generation mechanism without duplicating the logic.

The class is fully implemented in Core Java and does not require any external libraries. It defines a functional interface, ExistsByCode, that checks whether a specific shortcode already exists, and another interface, PutMapping, that stores a newly created mapping. This makes the Creator a generic tool that can take advantage of any implementation of UrlMappingStore . The following code snippet illustrates the structure:

javaCopy

publicfinalclassMappingCreatorimplementsHasLogger{privatefinalShortCodeGeneratorgenerator;privatefinalExistsByCodeexists;privatefinalPutMappingstore;privatefinalclockclock;privatefinalFunction<ErrorInfo,String>errorMapper;publicMappingCreator(ShortCodeGeneratorgenerator,ExistsByCodeexists,PutMappingstore,Clockclock,Function<ErrorInfo,String>errorMapper){this.generator=Objects.requireNonNull(generator);this.exists=Objects.requireNonNull(exists);this.store=Objects.requireNonNull(store);this.clock=Objects.requireNonNullElse(clock,Clock.systemUTC());this.errorMapper=Objects.requireNonNull(errorMapper);}

The complete creation process is then displayed in the create method. The process follows a precisely defined scheme: First, it is checked whether the user has entered an alias. If so, it is checked and normalised via the AliasPolicy’s validate method. If the alias is invalid, the creator creates an error object with HTTP and application code and returns it as a failure result. If no alias is available, the creator automatically generates a unique shortcode with the ShortCodeGenerator, checks for collisions, and repeats the process if necessary.

xmlCopy

public Result<ShortUrlMapping> create(String alias, String url) { logger().info("createMapping - alias='{}' / url='{}'", alias, url); final String shortCode; if (!isNullOrBlank(alias)) { var aliasCheck = AliasPolicy.validate(alias); if (aliasCheck.failed()) { var reason = aliasCheck.reason(); var reasonCode = switch (reason) { case NULL_OR_BLANK -> "ALIAS_EMPTY"; case TOO_SHORT -> "ALIAS_TOO_SHORT"; case TOO_LONG -> "ALIAS_TOO_LONG"; case INVALID_CHARS -> "ALIAS_INVALID_CHARS"; case RESERVED -> "ALIAS_RESERVED"; }; var errorJson = errorMapper.apply(new ErrorInfo("400", reason.defaultMessage, reasonCode)); return Result.failure(errorJson); } var normalized = normalize(alias); if (exists.test(normalized)) { var errorJson = errorMapper.apply(new ErrorInfo("409", "normalizedAlias already in use", "ALIAS_CONFLICT")); return Result.failure(errorJson); } shortCode = normalized; } else { String gen = normalize(generator.nextCode()); while (exists.test(gen)) { gen = normalize(generator.nextCode()); } shortCode = gen; } var mapping = new ShortUrlMapping(shortCode, url, Instant.now(clock), Optional.empty()); store.accept(mapping); return Result.success(mapping);}

This method shows the functional structure of the component as an example. The creator does not create side effects outside its intended interfaces. It communicates exclusively via the function objects exists and store. Error handling is fully integrated into the result tree, allowing a clean separation between success and failure paths. In this way, logic remains deterministic, testable, and reproducible.

With the introduction of MappingCreator, the codebase has been significantly streamlined. Both the InMemoryUrlMappingStore and the EclipseStoreUrlMappingStore now use the same generation logic, which substantially improves consistency and extensibility. The Creator is thus not only a refactoring result, but also a strategic architectural element that combines functional abstraction, type safety and test-driven development.

EclipseStore – The Persistent Foundation

After the internal generation logic has been unified with the MappingCreator, the implementation of the EclipseStoreUrlMappingStore now serves as the basis for permanent storage of the generated short URLs. This class replaces volatile storage with a fully persistent object model that maintains the application’s state across reboots. While the InMemory store only kept all mappings in a ConcurrentHashMap, in the EclipseStore they are stored within a serializable object, the so-called DataRoot. This structure serves as an anchor for all stored data elements, ensuring the entire object graph remains consistent at all times.

The central idea of the EclipseStore approach is to implement storage logic not via relational mappings but via object-oriented persistence. This preserves the application’s model in its entirety, without requiring object or database table conversions. The following example shows the definition of the DataRoot:

javaCopy

publicclassDataRootimplementsSerializable{@SerialprivatestaticfinallongserialVersionUID=1L;privatefinalMap<String,ShortUrlMapping>mappings=newConcurrentHashMap<>();publicMap<String,ShortUrlMapping>mappings(){returnmappings;}}

In the EclipseStoreUrlMappingStore, when the server starts, it checks whether a root structure already exists. If not, it is recreated and registered as the root node of the memory instance. The following snippet shows the relevant section from the constructor:

kotlinCopy

varstoragePath=Paths.get(storageDir);Files.createDirectories(storagePath);this.storage=EmbeddedStorage.start(storagePath);DataRootr=(DataRoot)storage.root();if(r==null){storage.setRoot(newDataRoot());storage.storeRoot();}

This initialisation ensures that the application has the same persisted state both during a cold boot and after a shutdown. All access to the mappings then takes place directly via the root object. When a new mapping is created, the store calls the storeMappingAndPersist method , which stores the object in the internal map and synchronises the memory.

javaCopy

privatevoidstoreMappingAndPersist(ShortUrlMappingm){vardataRoot=(DataRoot)storage.root();varmappings=dataRoot.mappings();mappings.put(m.shortCode(),m);storage.store(mappings);}

This immediate write process saves changes immediately. The EclipseStore automatically takes over transaction management at the object level and ensures atomic storage without requiring additional commit logic. This reduces the risk of errors and increases the traceability of the system’s status.

In addition, the EclipseStoreUrlMappingStore implements all the methods of the UrlMappingStore interface, including find, delete, count, and existsByCode. These methods operate directly on the object graph and use helper classes, such as UrlMappingFilterHelper, to efficiently execute queries and perform sorting. The key difference to the InMemory store is that changes to the data structure take effect immediately in persistent memory and are therefore available again when the application restarts.

The implementation of the EclipseStoreUrlMappingStore makes it clear that persistence was not designed as an afterthought, but as an integral part of the architecture. The entire data flow – from alias generation to MappingCreator to storage in DataRoot – follows a consistent design. This not only achieves persistence, but also anchors it conceptually: the memory is not an external component, but part of the living object model of the application.

Additional improvements in the core

Parallel to the introduction of the EclipseStore and the standardisation of the mapping logic, the central helper classes in the core module have also been further developed to ensure more consistent data processing. The focus was not on expanding the range of functions, but on making internal processes more precise. In particular, JsonUtils, UrlMappingFilterHelper, and the internal validation logic of the AliasPolicy have been explicitly revised to improve integration between the REST layer, the UI, and memory.

The JsonUtils class has been fully refactored to improve the security and robustness of data object serialisation and deserialisation. Instead of resorting to external parsers or frameworks, the class implements a well-defined strategy for simple objects and for nested structures. The focus is on stability, readability and deterministic processing. A central component is the toJsonListingPaped method, which is used for tabular representation in the administration interface. It generates a JSON array from a list of mapping objects, including metadata such as the total number and paging information.

xmlCopy

public static String toJsonListingPaged(List<ShortUrlMapping> list, int total) { var sb = new StringBuilder(); sb.append('{'); sb.append("\"total\":").append(total).append(','); sb.append("\"items\":["); for (int i = 0; i< list.size();i++){sb.append(toJson(list.get(i)));if(i<list.size()-1)sb.append(',');}sb.append("]}");returnsb.toString();}

This method exemplifies the class’s philosophy: complete control over serialisation and field order. This guarantees that the generated JSON structures correspond precisely to the expected format – an important point when the backend and UI are developed independently of each other. The use of StringBuilder is not a stylistic device, but a deliberate design to minimise garbage objects under high request load.

Another central component is the UrlMappingFilterHelper, which handles dynamic processing of filter and sorting parameters for REST endpoints. This class abstracts the transformation of the user-entered filters into internal predicate objects, which are then applied to the stored mappings at runtime. The implementation supports flexible queries without requiring specialised query languages or parsers. The following excerpt shows the method that evaluates a URL mapping based on several parameters:

javaCopy

publicstaticbooleanmatches(ShortUrlMappingm,UrlMappingListRequestreq){if(req.codePart()!=null&&!m.shortCode().contains(req.codePart()))returnfalse;if(req.urlPart()!=null&&!m.originalUrl().contains(req.urlPart()))returnfalse;if(req.from()!=null&&m.createdAt().isBefore(req.from()))returnfalse;if(req.to()!=null&&m.createdAt().isAfter(req.to()))returnfalse;returntrue;}

This compact logic ensures that filter requests are processed directly at the object level. The system does not have to generate intermediate representations or convert data. The result is efficient, storage-level filtering that scales to large datasets. In combination with the store’s advanced counting methods, it provides a high-performance foundation for complex search and administrative functions.

This area is rounded off by the extension of the AliasPolicy, in particular by the introduction of the helper method failed(). This is used for precise error evaluation and simplifies the control logic in components such as the MappingCreator. Instead of laboriously checking validation results, the developer can directly ask if a validation has failed, and then take fine-grained action based on the return value. This not only improves the readability of the codebase but also the overall system’s error resistance.

The adjustments above make it clear that there is a well-thought-out concept behind the supposed auxiliary classes. They are not marginal components, but supporting pillars of data consistency and system stability. Through targeted optimisations and a clear methodological framework, a core module is created that serves as the basis for the project’s further development, both functionally and architecturally.

Before & After – Impact on the developer experience

With the integration of EclipseStore and the unification of generation logic via MappingCreator, the developer experience in this project has fundamentally changed. What began as a loose interplay of individual components has developed into a precisely orchestrated system that convinces both in its architecture and maintainability. This is particularly evident in the reduction of redundancies, improved testability and increased transparency in the code flow.

In the past, central operations – especially the creation and verification of new mappings – were scattered across multiple classes. The InMemory store contained its own validation and checking routines, while later extensions had to reimplement the same logic for persistent storage. With the MappingCreator, this process has now been transformed into a clearly defined, reusable unit. This not only eliminates duplicate code but also the risk of different implementations delivering divergent results. The following excerpt shows an example of how the creation of a mapping was previously realised within the store:

javaCopy

publicShortUrlMappingcreate(Stringalias,Stringurl){StringshortCode=alias!=null?alias:generator.nextCode();if(map.containsKey(shortCode)){thrownewIllegalStateException("Alias already exists: "+shortCode);}varmapping=newShortUrlMapping(shortCode,url,Instant.now(),Optional.empty());map.put(shortCode,mapping);returnmapping;}

This logic was functionally correct, but inflexible. Neither validation nor error objects nor alias rules were built in, and there was no uniform handling of failures or conflicts. With the introduction of MappingCreators and the centralised error structure, this process has been redesigned. The result is a deterministic, controlled, and traceable generation of short URLs, in which every step – from input to persistence – occurs via clearly defined interfaces.

The second significant difference concerns how the memory is addressed. Previously, the entire application was designed for volatile data retention, which meant that each reboot resulted in a complete loss of the stored mappings. With the EclipseStore, this state has been fundamentally changed. The following code shows how the Store initialises its persistent database when the application starts:

kotlinCopy

varstoragePath=Paths.get(storageDir);Files.createDirectories(storagePath);this.storage=EmbeddedStorage.start(storagePath);DataRootr=(DataRoot)storage.root();if(r==null){storage.setRoot(newDataRoot());storage.storeRoot();}

These few lines mark the crucial difference between temporary and permanent system states. The DataRoot’s data structure is loaded at startup, changes are immediately synchronised, and the state is maintained across reboots. From a developer’s point of view, this means that unit tests, integration tests and runtime behaviour can now consistently access the same database. The difference between development and production disappears because both use the same data flow and persistence logic.

This standardisation also directly affects the user interface. The Vaadin components, such as OverviewView and StoreIndicator, now obtain their data via clearly defined interfaces that are decoupled from the persistence layer. State changes of the store or new mappings trigger events that are consumed by the UI components, without explicit dependencies. This creates a consistent, reactive system that works coherently from the data source to the surface.

This development has fundamentally changed how developers interact with code. Bugs are easier to reproduce, tests are more stable, and extensions can be modular. The code is no longer a collection of independent methods, but a structured system in which each component has its place and responsibility.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-persistence-part-01/Thu, 04 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-persistence-part-01/**Visible change: When the UI shows the memory state With the end of the last day of the Advent calendar, our URL shortener was fully functional: the admin interface could filter, sort, and display data page by page – performant, cleanly typed, and fully implemented in Core Java. But behind this surface lurked an invisible problem: All data existed only in memory. As soon as the server was restarted, the entire database was lost.<![CDATA[

**Visible change: When the UI shows the memory state

With the end of the last day of the Advent calendar, our URL shortener was fully functional: the admin interface could filter, sort, and display data page by page – performant, cleanly typed, and fully implemented in Core Java. But behind this surface lurked an invisible problem:All data existed only in memory. As soon as the server was restarted, the entire database was lost.

The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-02

Here’s the screenshot of the version we’re going to implement now.

**Why pure in-memory data is reaching its limits

This volatility was acceptable for early testing, but as the functionality depth increased, clear limits became apparent. In-memory data has three fundamental drawbacks:

• Volatility: Each reboot resets the state – a behaviour that leads to unusable test and production data in real admin systems.
• Missing history: Statistical evaluations or comparisons across sessions are impossible because the state is not persisted over time.
• Invisible state: The user does not see in the UI whether they are working with persistent or volatile memory.

This meant that the application remained technically correct, but operationally incomplete. A real administration interface not only needs functions, but alsoreliability and transparency about the system status.

**Motivation for the introduction of EclipseStore

This is whereEclipseStore comes into play.The project aims to make a seamless transition from in-memory data structures to persistent object graph storage –without ORM, without database, without a break in the architecture.
This makes it perfectly in line with the philosophy of this project: type safety, simplicity and complete control over the data flow.

EclipseStore stores the same object graph that we already use in memory permanently on disk.No mapping, no SQL, no external dependency injection layer – the application remainspure Java. By integrating EclipseStoreUrlMappingStore, the shortener can run in two modes:

• InMemory for quick tests and volatile sessions,
• EclipseStore for productive, persistent runtimes.

This duality not only enables real operational data, but also a clean transition between the development and production environments – an essential aspect if you want to systematically upgrade applications without rewriting them.

**Goal: Permanent storage and live status in the UI

But persistence alone was not the goal of this chapter; it was at least as necessary that the condition becamevisible. If the server is in persistent mode, the user interface should show this immediately – preferably in real time.

This is precisely where the newstatus display in the MainLayout comes in: A small icon with colour feedback (green, blue, red) signals whether the system is currently volatile, persistent or unreachable. Underneath are mechanisms that cyclically query the server state, distribute events, and automatically update the UI when the mode changes.

This merges the backend and frontend into a standard, reactive system: The interface no longer becomes just a window on stored data, buta visible mirror of the system state.
This coupling of persistence and UI status forms the foundation for all upcoming enhancements – especially for security, rights management and operational diagnostics.

**The new MainLayout – context and status at a glance

After we have created the theoretical basis in the previous chapter and understood why volatile InMemory data is not sufficient and how EclipseStore enables persistence, we now turn our attention to the user interface. Every architectural decision becomes tangible only when it is reflected in the UI, both visually and functionally.

The most visible expression of this change is in theMainLayout. With the introduction of EclipseStore, it received a new, subtle, but highly effective addition: theStoreIndicator. This small but central component is located in the application header and shows at a glance whether the shortener is currently using a volatile InMemory store or a persistent EclipseStore.

This transforms a previously static header into a dynamic control centre that is not only navigative, but also informative – a visual link between the user interface and the system state.

**Architectural Concept – Visibility as a System Function

In the original Vaadin interface, the header layout consisted only of a title and the navigation structure. The new component now expands this area: between the title“URL Shortener” and the menu, an icon with a label appears, whose colour and text change depending on the memory status.

The concept behind it is deliberately kept minimalist: Status information should not be displayed via dialogues or messages, but should bepermanently present , similar to a status bar in professional administration tools.

**Technical integration

The integration was carried out via the existing MainLayout, which serves as the central layout template in Vaadin. There, a StoreIndicator is added when building the header :

kotlinCopy

varadminClient=AdminClientFactory.newInstance();varstoreIndicator=newStoreIndicator(adminClient);storeIndicator.getStyle().set("margin-left","auto");AlignRightHorizontalLayoutheaderRow=newHorizontalLayout(toggle,appTitle,newSpan(),storeIndicator);headerRow.setWidthFull();headerRow.setAlignItems(FlexComponent.Alignment.CENTER);

The StoreIndicator internally uses an AdminClient that periodically queries the server endpoint /store/info. This provides JSON information about the current mode, the number of saved mappings, and the time of server start.

**UI design and colour code

For better orientation, a three-level colour scheme has been introduced:

• 🟢EclipseStore active: Persistent persistence, data is written to disk.
• 🔵In-Memory Mode: Volatile memory, suitable for testing and fast development cycles.
• 🔴Error condition: Connection unreachable or response invalid.

This colour system is implemented consistently with Vaadin’s Lumo design system and ensures that the status indicator blends harmoniously into the UI.

**Effect on user experience

The effect is subtle but profound:
Users can immediately recognise whether they are working with a persistent or a temporary system. At the same time, there is greater trust in the application, as the interface now provides transparency into the system’s status. This aspect is often neglected in professional web applications.

In the next chapter, we will look at how theStoreIndicator works technically: from the polling mechanism to the event architecture to its self-updating in the UI.

**Live Status Component: The StoreIndicator in Detail

After the visible integration in the MainLayout, let’s now turn our attention to the StoreIndicator’s actual functionality**.** This central component displays the system’s state in real time. While the header in the UI provides the context, the StoreIndicator is the heart that makes data movement visible and gives immediate feedback to the user.

**Architecture and Purpose

The StoreIndicator was designed as a standalone Vaadin component that extends HorizontalLayout. It combines logic, display and event control in a clearly defined unit. Their goal: to regularly retrieve the current memory state (InMemory, EclipseStore or error state) and display it visually.

To do this, the component uses periodic polling, which queries the server endpoint /store/info via an AdminClient . The response includes metadata such as memory mode, number of saved mappings, and the time of server startup. Based on this data, the display changes dynamically.

**Lifecycle in the Vaadin UI

When the user interface is attached (onAttach), polling is activated. The UI retrieves the current system information at fixed intervals (every 10 seconds). When removing (onDetach), this background activity is terminated again, keeping the application resource-efficient.

javaCopy

@OverrideprotectedvoidonAttach(AttachEventattachEvent){   refreshOnce();   UIui=attachEvent.getUI();   ui.setPollInterval(10000);every10seconds   ui.addPollListener(e->refreshOnce());}

**Communication with the backend

Communication between the frontend and the backend occurs via the AdminClient. This calls the JSON endpoint /store/info and converts the response to a StoreInfo object:

**StoreInfo info = adminClient.getStoreInfo();

**boolean persistent = “EclipseStore”.equalsIgnoreCase(info.mode());

On this basis, color, icon and text are updated. The visual status is based on three states:

• 🟢EclipseStore active – persistent storage, data is stored on disks.
• 🔵InMemory mode – volatile storage, ideal for testing and development.
• 🔴Error condition – Connection lost or server unreachable.

These states are immediately visible in the UI and follow the colour values of Vaadin’s Lumo theme, which ensures a consistent design.

**Event-Based Update

In addition to polling, the StoreIndicator relies on an event-based notification system. Via StoreEvents.publish(), it sends a global event whenever the storage state changes. Components such as the OverviewView can subscribe to this event and automatically reload when changes are made.

**StoreEvents.publish(new StoreConnectionChanged(newMode, info.mappings()));

This loosely coupled event system avoids direct dependencies between UI components and allows for flexible extensibility – a pattern that can also be applied to other UI areas.

The StoreIndicator, which displays the system’s state in real time, operates resource-efficiently and enhances user confidence in the application. It turns the abstract term “persistence mode” into a visual experience – a perfect example of how backend information can be elegantly integrated into the UI.

Cheers Sven

]]>EclipseStoreJavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-filter-search-part-02/Wed, 03 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-filter-search-part-02/In the previous part, we looked at the implementation on the server side. This part is now about the illustration on the user page. The source code for the initial state can be found on GitHub under https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-00 .<![CDATA[

In the previous part, we looked at the implementation on the server side. This part is now about the illustration on the user page.

The source code for the initial state can be found on GitHub underhttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-00 .

The following screenshot shows this state of development.

The focus of this Advent calendar day is on the introduction of targetedfiltering, search and paging functionality on the UI side. The goal is to extend the existing “Overview” view so that users can search specifically for specific shortcodes or URL fragments, restrict time periods and retrieve the results page by page. All the technical foundations for this have been laid in the previous part.

The source code for today’s articles can be found on Github underhttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-01 .

The following screenshot shows the state of development that we will reach today.

Integration into the Vaadin UI

After implementing the filtering and paging logic on the server and client side, the focus is now on integration into the user interface. The Vaadin UI acts as the visible layer of the system and is crucial to the user experience. The aim of this chapter is to show how the new functions – filtering, searching and page by page – are brought together in a modern, reactive interface.

In contrast to the previous parts, which primarily described technical structures, the focus here is on theinteraction between user and system. TheOverviewView serves as a central view through which users communicate directly with the API without having to know technical details.

The following sections show how the new API endpoints and client methods create a dynamic yet intuitive interface – from data binding and filter logic to visual fine-tuning and ergonomic design.

OverviewView – Filtering, Searching, and Paging in the Vaadin UI

With the new server and client functions, the basis for an interactive user interface is now available. This chapter describes the revisedOverviewView – the central view of the Vaadin application in which all existing short links can be listed, filtered and managed.

The original version of the View only showed a complete list of all mappings. In the new version, it has been expanded into adynamic, filterable and page-based interface that communicatesdirectly with the endpoints/listand/list/count.

Structure of the user interface

TheOverviewView combines several input elements to capture search and filter criteria:

• Text fields for shortcode and URL substrings (codePart,urlPart)
• Case-sensitive checkboxes (codeCase,urlCase)
• Date and time selection for time periods (fromDate,toDate)
• Dropdowns for sorting (sortBy,dir)
• Page size via a numeric input field (pageSize)

In addition, navigation buttons (Prev /Next) enable side control. All input is transferred to aUrlMappingListRequest object, which is passed to theURLShortenerClientwhen the view is refreshed .

Central logic: DataProvider

The data supply to the grid is provided by aCallbackDataProvider, which reacts dynamically to user interactions. This calls two methods of the client in the background:

1. **list()** – loads the actual records of the current page.
2. **listCount()** – determines the total number to control page navigation.

A simplified excerpt of the initialization:

javaCopy

dataProvider=newCallbackDataProvider<>(q->{varreq=buildFilter(currentPage,pageSize.getValue());returnurlShortenerClient.list(req).stream();},q->{varbaseReq=buildFilter(null,null);totalCount=urlShortenerClient.listCount(baseReq);refreshPageInfo();returntotalCount;});

This architecture enables the View to react immediately to any change in the filter without having to reload the entire page. The user can conveniently browse through the results, adjust filters or change time periods – the data is loaded live in each case.

UX Improvements

In addition to the functional enhancement, the user experience has also been improved:

• Responsive layouts: All filter elements arrange themselves flexibly, even with smaller window sizes.
• Automatic page navigation: ThePrev andNext buttons are automatically deactivated depending on the context.
• Date-time combination: Separate DatePicker and TimePicker fields keep input precise and intuitive.

These innovations lead to considerable added value in the daily use of the application: Instead of static tables, there is now an interactive, reactive interface that is both more performant and more ergonomic.

In the next section, we’ll look at the implementation of the paging and filtering logic in detail, including thebuildFilter() method and how to handle the parameters in the UI.

Implementation details of the filter logic – buildFilter() and data binding

ThebuildFilter() method forms the heart of the new filter mechanics in the Vaadin interface. It collects all of the user’s input—such as text fields, checkboxes, dates, and sorting options—and transfers them to aUrlMappingListRequest object. This request object is then passed on to the client, which uses it to generate the appropriate query parameters for the server.

Structure of the buildFilter() method

The focus is on the builder structure of theUrlMappingListRequest. The method first reads all UI components, checks for empty fields, and creates a consistent filter object from them:

javaCopy

privateUrlMappingListRequestbuildFilter(Integerpage,Integersize){UrlMappingListRequest.Builderb=UrlMappingListRequest.builder();if(codePart.getValue()!=null&&!codePart.getValue().isBlank()){b.codePart(codePart.getValue());}b.codeCaseSensitive(Boolean.TRUE.equals(codeCase.getValue()));if(urlPart.getValue()!=null&&!urlPart.getValue().isBlank()){b.urlPart(urlPart.getValue());}b.urlCaseSensitive(Boolean.TRUE.equals(urlCase.getValue()));if(fromDate.getValue()!=null&&fromTime.getValue()!=null){varzdt=ZonedDateTime.of(fromDate.getValue(),fromTime.getValue(),ZoneId.systemDefault());b.from(zdt.toInstant());}elseif(fromDate.getValue()!=null){varzdt=fromDate.getValue().atStartOfDay(ZoneId.systemDefault());b.from(zdt.toInstant());}if(toDate.getValue()!=null&&toTime.getValue()!=null){varzdt=ZonedDateTime.of(toDate.getValue(),toTime.getValue(),ZoneId.systemDefault());b.to(zdt.toInstant());}elseif(toDate.getValue()!=null){varzdt=toDate.getValue().atTime(23,59).atZone(ZoneId.systemDefault());b.to(zdt.toInstant());}if(sortBy.getValue()!=null&&!sortBy.getValue().isBlank())b.sort(sortBy.getValue());if(dir.getValue()!=null&&!dir.getValue().isBlank())b.dir(dir.getValue());if(page!=null&&size!=null){b.page(page).size(size);}returnb.build();}

The method works strictly defensively: empty fields are ignored, optional values are only set if they actually exist. The result is always a valid, well-defined request object.

Data flow between UI and server

The generated filter objects are passedto theURLShortenerClient via the CallbackDataProvider. This generates the query string for the HTTP request. This creates a clear, linear chain:

javaCopy

UI→UrlMappingListRequest→client→/listendpoint→filter→results

Every change in the UI – be it a new search, a changed page size or a date filter – automatically triggers a refresh. The Grid component then updates itself with the newly filtered data.

Advantages of encapsulation

This structure offers several advantages:

• Reusability:buildFilter() encapsulates all UI logic in one method.
• Testability: The result can be easily tested in unit tests without the need for Vaadin-specific classes.
• Extensibility: New filter fields can be easily added as long as they are stored in the builder.

This clean decoupling keeps the Vaadin interface clutter-free, while at the same time achieving deep integration with the API. ThebuildFilter() method is thus the central node between user interaction and server-side data logic.

Paging and user interaction – control, display, feedback

The revisedOverviewView offers intuitive control for page-by-page navigation through large result sets. The aim is for users to quickly seewhere they are in the results list,how many elements exist in total andwhich interactions are currently possible.

Paging elements in the UI

• Buttons:Prev (to the previous page) andNext (to the next page).
• Page size:IntegerField pageSize with min/max limits and step buttons.
• Status display:pageInfo shows e.g. “Page 3 / 12 • 289 total”.

These elements are directly linked to the DataProvider and are updated after each query.

Changing sides and limiting

Clicking onPrev/Next adjusts the current page, then the View reloads the data:

javaCopy

prevBtn.addClickListener(e->{if(currentPage>1){currentPage--;refresh();}});nextBtn.addClickListener(e->{intsize=Optional.ofNullable(pageSize.getValue()).orElse(25);intmaxPage=Math.max(1,(int)Math.ceil((double)totalCount/size));if(currentPage<maxPage){currentPage++;refresh();}});

The page size directly affects the calculation ofmaxPage. Changes to thepageSize field reset the current page to1 and trigger a refresh:

javaCopy

pageSize.addValueChangeListener(e->{currentPage=1;grid.setPageSize(Optional.ofNullable(e.getValue()).orElse(25));refresh();});

Status Indicator Update

TherefreshPageInfo() method synchronizes buttons and display with the current data state:

javaCopy

privatevoidrefreshPageInfo(){intsize=Optional.ofNullable(pageSize.getValue()).orElse(25);intmaxPage=Math.max(1,(int)Math.ceil((double)totalCount/size));currentPage=Math.min(Math.max(1,currentPage),maxPage);pageInfo.setText("Page "+currentPage+" / "+maxPage+" • "+totalCount+" total");prevBtn.setEnabled(currentPage>1);nextBtn.setEnabled(currentPage<maxPage);}

This prevents the navigation from getting into invalid states (e.g.Prev on page 1 orNext after the last page).

Interaction with list() and listCount()

• Count (**listCount**): Determines the total amount (totalCount)without transferring data that does not need to be displayed.
• Data (**list**): Loads only the currently visible subset (based on page/size and current filters).

The combination minimizes network load and ensures consistently fast response times.

UX details and fault tolerance

• Disable logic: Buttons are automatically (de)activated depending on currentPage/maxPage.
• Error handling: Network or server errors areimmediately visible viaNotification.show(“Loading failed”).
• Reset behavior: Thereset button sets filters to default values,currentPage to 1 and triggersrefresh().

These mechanisms create a reactive, robust paging experience that remains clear even with very large amounts of data and provides clear feedback to the user.

UX & styling – visual and ergonomic refinements

Now that the functionality of the filtering, search and paging mechanisms has been established, this chapter focuses on the fine-tuning of the user interface. The aim is to improve the user experience of the Vaadin interface without changing the technical structure. The focus is on clarity, responsiveness and consistent visual feedback.

Basic design principles

1. Clarity over complexity – Each function (e.g., filtering, sorting, paging) should be visually recognizable without overwhelming the user with options.
2. Visual grouping – Logically related items are grouped into horizontal or vertical layouts (e.g., time range, sorting options, search fields).
3. Consistent feedback – Every user action (e.g., changing a filter, changing pages, deleting action) receives direct visual or textual feedback.

Exemplary adjustments

1. Layout structure

The input elements for filters and paging have been divided into several layout lines:

kotlinCopy

varfilterRow=newHorizontalLayout(codePart,codeCase,urlPart,urlCase,fromGroup,toGroup,sortBy,dir);filterRow.setDefaultVerticalComponentAlignment(Alignment.END);varpagingRow=newHorizontalLayout(prevBtn,nextBtn,pageInfo,pageSize);pagingRow.setDefaultVerticalComponentAlignment(Alignment.CENTER);add(filterRow,pagingRow,grid);

This division semantically separatesfilters andnavigation and thus improves visual orientation.

2. Color and Theme Consistency

The buttons now use Vaadin’s own theme variants to convey semantic meaning:

javaCopy

searchBtn.addThemeVariants(ButtonVariant.LUMO_PRIMARY);resetBtn.addThemeVariants(ButtonVariant.LUMO_CONTRAST);deleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR,ButtonVariant.LUMO_TERTIARY);

This immediately signals to the user which actions are primary, neutral or destructive.

3. Responsive behavior

By usingsetWrap(true) and percentage widths, the layout is also optimally displayed on smaller screens. Filter elements automatically wrap into new lines without disturbing the overall impression.

javaCopy

filterRow.setWrap(true);filterRow.setWidthFull();

4. Interactive status messages

Error and success messages are communicated viaNotification.show(). In addition, the affected element can optionally be visually marked – for example, by means of temporary color changes or tooltips.

javaCopy

Notification.show("Short link deleted.");

Bringing together function and aesthetics

These design and ergonomic fine-tunes make theOverviewView a reactive tool for everyday use. The application not only appears technically sophisticated, but also visually conveys the demand for precision and quality.

This completes the implementation of the UI for the new filter and search logic – a basis on which later extensions such as caching, auto-suggest or extended sorting logics can be built.

Conclusion and outlook

With the implementation of the filter, search and paging functions, the first milestone of the Advent calendar project has been reached. A static overview has become a dynamic, interactive system that allows data to be accessed in a targeted and efficient manner. The basic idea of the project – a solution implemented entirely in Core Java without external frameworks – has been consistently retained.

Review

This day showed how a clear separation of responsibilities and clean API design can create a solid basis for more complex functionalities:

• Server-side : Introduction of theUrlMappingFilter and expansion of theInMemoryUrlMappingStore for structured queries.
• Client-side : Extension of theURLShortenerClient with type-safe methods for filtering and counting.
• UI side : Integration of the new features into the Vaadin-based interface with reactive paging, intuitive navigation and clear user guidance.

The result is a consistent overall system that remains modular, expandable and testable.

Cheers Sven

]]>JavaServersideVaadinhttps://svenruppert.com/posts/advent-calendar-2025-filter-search-part-01/Tue, 02 Dec 2025 07:05:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/advent-calendar-2025-filter-search-part-01/With the Vaadin interface described in [Part III], our URL shortener has a fully functional administration console available for the first time. It allows viewing existing short links in tabular form and managing them manually. But after just a few dozen entries, a clear limit becomes apparent: displaying all saved mappings is neither performant nor user-friendly. An efficient shortener must be able to scale – not only when generating, but also when searching through its data.<![CDATA[

With the Vaadin interface described in[Part III], our URL shortener has a fully functional administration console available for the first time. It allows viewing existing short links in tabular form and managing them manually. But after just a few dozen entries, a clear limit becomes apparent: displaying all saved mappings is neither performant nor user-friendly. An efficient shortener must be able to scale – not only when generating, but also when searching through its data.

1. Architecture Extension
2. Server-side changes
   1. UrlMappingFilter – the new filter model
   2. ListHandler - Extended GET Endpoint (/list)
   3. ListCountHandler - Lightweight counting endpoint (/list/count)
   4. InMemoryUrlMappingStore – Filtering, sorting, and paginating
   5. QueryUtils – Parsing and Normalising Query Parameters
3. Introduction to API Endpoints
   1. Structure of the new endpoints
      1. Compatibility and stability
   2. Supported Parameters
      1. Interaction of the parameters
4. Client-side extensions
   1. UrlMappingListRequest – Filter and Paging Request Builder
5. Extension of the URLShortenerClient – Filter and Count
   1. New methods
   2. Significance of enlargement
   3. Backward compatibility
6. Client Test Coverage – URLShortenerClientListTest
   1. Paging and sorting tests
   2. Aim of the tests

The source code for this status can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-00.

The screenshot below shows the status we start with.

The focus of this first day of the Advent calendar is therefore on introducing targetedfiltering, search, and paging functionality. The goal is to extend the existing “Overview” view so that users can search for specific shortcodes or URL fragments, restrict time periods, and retrieve results page by page. All the technical foundations for this have been laid in the previous parts:

1. Part I – Short links, clear architecture : A detailed introduction to the motivation, architecture and data model of the project. The article shows how to develop a URL shortener entirely in Core Java, along with the design principles behind it. (https://svenruppert.com/2025/06/10/short-links-clear-architecture-a-url-shortener-in-core-java/ )
2. Part II – Deepening the Server Components : This article explores REST handlers, validation, security considerations, and the interaction between the core module and the server layer. (https://svenruppert.com/2025/06/20/part-ii-urlshortener-first-implementation/)
3. Part III – The Web UI with Vaadin Flow: This shows how the user interface is structured, how lists, forms, dialogues and validations work, and how Vaadin Flow enables a modern UI without an additional JavaScript stack. (https://svenruppert.com/2025/08/15/part-iii-webui-with-vaadin-flow-for-the-url-shortener/)

The source code for today’s article can be found on Github underhttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-01.

The following screenshot shows the development status shown in it.

It is important that wedo not introduce framework magic , but consistently expand the existing system with the JDK’s on-board resources. Every new class, additional parameter, and API detail follows the same principles as before: clarity, type safety, and transparency. The changes are deliberately incremental – they do not replace existing functions, but expand them in a controlled manner.

Below, we’ll take a step-by-step look at how this new layer was introduced: from the server-side filtering model to the extended REST endpoint and integration with the Vaadin UI. All new code sections are explicitly highlighted so that the differences to the previous parts can be clearly understood.

Architecture Extension

The module structure established in the previous parts is completely retained:core ,api ,client andui-vaadin continue to form the central layers of the system. However, there is a new logical level between the REST endpoint and the data storage: thefilter model. It allows you to describewhat data the server is supposed to deliver precisely – and no longer justthat it returns all existing mappings.

This architectural principle follows the responsibility separation principle described in [Part II]: Each layer should do precisely what is within its responsibility. The REST API receives requests, converts them into type-safe filter objects, and passes them to the store. The store, in turn, does the actual searching, sorting, and page layout. The UI only uses filtered results, which keeps it lightweight and responsive.

The new architecture can therefore be understood as a small but decisive extension of the previous data flow:

[UI] → [Client] → [API] → [Filter Model] → [Store]

Previously, communication between the UI and the API was mainly in the form of complete data lists; now, queries with parameters are transmitted in a targeted manner. These parameters describe search patterns, time periods, sorting, and paging information. The result is a much more flexible and high-performance interaction that fits seamlessly into the existing system.

Conceptually, care was taken to ensure that all new components could be integratedincrementally. No existing function will be changed or replaced. Instead, clients that continue to use the legacy endpoints (e.g. /list/all) do so unchanged. New clients – such as the revised OverviewView – on the other hand, immediately benefit from the extended filter options.

This decoupling not only ensures stability during operation but also forms the basis for future extensions, such as persistent filters or server-side search indexes. The following sections now show in concrete terms how this architecture was technically implemented.

Server-side changes

After the architecture has been defined stably in the previous parts, this section focuses on extending the server-side. The goal is to complement the existing REST endpoints so that they can deliver filtered, sorted, and paginated results in a targeted manner. The previous URL shortener structure has not changed, but has been expanded into clearly defined modules.

The following subchapters describe the main new building blocks – from the centralUrlMappingFilter to the revised handlers and the helper classes that handle parsing and data processing.

UrlMappingFilter – the new filter model

To enable targeted searches, a new class has been introduced: UrlMappingFilter. It forms the heart of the server-side extension and defines all the parameters that precisely describe a query – from text fragments and date ranges to sorting and paging.

The structure consistently follows thetype-safe API construction principle established in [Part II]. Instead of evaluating unstructured query parameters directly in the handler, UrlMappingFilter encapsulates all possible filter options in a clearly defined object. This keeps the REST logic lean and the code easier to test.

A typical filter object might look like this:

kotlinCopy

varfilter=UrlMappingFilter.builder().codePart("ex-").urlPart("docs").createdFrom(Instant.parse("2025-10-01T00:00:00Z")).limit(25).sortBy(SortBy.CREATED_AT).direction(Direction.DESC).build();

Here, only the parameters relevant to a specific query are provided as examples. Unset values remain null or empty, which makes the builder particularly flexible. The class itself isimmutable – once it has been created, it cannot be changed.

The most important features at a glance:

• codePart, urlPart: Text-based filtering (substrings, optional case-sensitive)
• createdFrom, createdTo: Time constraint of the result
• offset, limit: Paging control (start index and number of records)
• sortBy, direction: Sort criteria (e.g. CREATED_AT or SHORT_CODE)

This clear structure allows the store to apply filters efficiently and easily integrate future extensions (such as status or user). It does not replace the existing search mechanism, but expands it modularly. Existing functions such as findAll() are retained.

The use of an explicit builder pattern ensures that filter objects can only be created in valid combinations. This principle has already proven itself with the ShortenRequest and ShortUrlMapping classes. In addition, consistent use of optional and explicit data types prevents empty strings or incorrect values from interfering with the filtering process.

This provides a basis for receiving requests in a structured format via the REST API and forwarding them to the store in a targeted manner. The following section shows how these filter objects are used within the ListHandler .

ListHandler - Extended GET Endpoint (/list)

The previous implementation of the ListHandler in [Part II] primarily returned all stored mappings as a static list. As the data stock grew, this was neither performant nor differentiated enough. As part of the new filter architecture, the handler has therefore been fundamentally expanded –without changing its previous endpoints. Existing calls such as /list/all or /list/expired will continue to work unchanged.

The new code path now recognises requests to the /list endpoint and interprets the query parameters. These are transferred to a UrlMappingFilter object and then passed to the store. The store then returns the filtered and sorted subset.

An excerpt from the new method:

javaCopy

privateStringlistFiltered(HttpExchangeexchange){varquery=parseQueryParams(exchange.getRequestURI().getRawQuery());intpage=parseIntOrDefault(first(query,"page"),1);intsize=clamp(parseIntOrDefault(first(query,"size"),50),1,500);intoffset=Math.max(0,(page-1)*size);varsortBy=parseSort(first(query,"sort"));vardir=parseDir(first(query,"dir"));booleancodeCase=Boolean.parseBoolean(first(query,"codeCase"));booleanurlCase=Boolean.parseBoolean(first(query,"urlCase"));varfilter=UrlMappingFilter.builder().codePart(first(query,"code")).codeCaseSensitive(codeCase).urlPart(first(query,"url")).urlCaseSensitive(urlCase).createdFrom(parseInstant(first(query,"from"),true).orElse(null)).createdTo(parseInstant(first(query,"to"),false).orElse(null)).offset(offset).limit(size).sortBy(sortBy.orElse(null)).direction(dir.orElse(null)).build();inttotal=store.count(filter);varresults=store.find(filter);varitems=results.stream().map(m->toDto(m,Instant.now())).toList();returnJsonUtils.toJsonListingPaged("filtered",items.size(),items,page,size,total,sortBy.orElse(null),dir.orElse(null));}

This method illustrates how cleanly the new filter logic integrates into the existing retailer. Instead of returning a complete list, it now creates apaged response object that contains additional metadata:

jsonCopy

{"mode":"filtered","page":2,"size":25,"total":123,"sort":"createdAt","dir":"desc","count":25,"items":[{...},{...}]}

This format offers two decisive advantages: On the one hand, the UI can work specifically with pagination, and on the other hand, API clients can be extended by further parameters in the future without breaking existing structures.

The trader must continue to pay attention to backward compatibility. All older endpoints are registered in the same context and continue to return complete JSON lists. Only /list uses the extended schema.

The ListHandler thus becomes the central intermediary between the API request and data filtering – it serves as the “translator” between HTTP parameters and the internal filter model. The following section shows how the supplemental counting endpoint /list/count uses the same logic for pure quantity queries.

ListCountHandler - Lightweight counting endpoint (/list/count)

In parallel with the extended /list endpoint, an additional handler, ListCountHandler, has been introduced. Its task is to provide only the total number of hits for a given filter configuration – without returning the actual data sets. This separation was deliberately chosen to make paging operations in the UI efficient.

In contrast to ListHandler, the counting endpoint does not return complete JSON objects with shortcodes and URLs, but only a compact counter value. This allows the client to determine in advance how many pages a query contains before loading individual data pages.

An excerpt from the implementation shows the simplicity of the approach:

xmlCopy

@Overridepublic void handle(HttpExchange exchange) throws IOException { if (!" GET".equalsIgnoreCase(exchange.getRequestMethod())) { exchange.sendResponseHeaders(405, -1); return; } Map<String,List<String>> q = parseQuery(Optional.ofNullable(exchange.getRequestURI().getRawQuery()).orElse("")); UrlMappingFilter filter = UrlMappingFilter.builder() .codePart(first(q, "code")) .codeCaseSensitive(bool(q, "codeCase")) .urlPart(first(q, "url")) .urlCaseSensitive(bool(q, "urlCase")) .createdFrom(parseInstant(first(q, "from")).orElse(null)) .createdTo(parseInstant(first(q, "to")).orElse(null)) .build(); int total = store.count(filter); byte[] body = ("{\"total\":" + total + "}").getBytes(StandardCharsets.UTF_8); exchange.getResponseHeaders().add("Content-Type", "application/json; charset=utf-8"); exchange.sendResponseHeaders(200, body.length); try (OutputStream os = exchange.getResponseBody()) { os.write(body); }}

The method remains minimalist: no paging, no sorting, no additional data. It uses the same parameters as the ListHandler, so both endpoints behave consistently. The only difference is in the return format – a deliberate design forefficiency and clear accountability.

An example of a typical client request:

GET /list/count?code=ex-&url=docs

→ { “total”: 42 }

The client (e.g. the Vaadin UI) can use this information to calculate the maximum number of pages and dynamically enable or disable the navigation buttons. This prevents unnecessary data transfer when the user only wants to knowhow many entries a filter currently delivers.

This lean endpoint lays the foundation for high-performance paging. The next step shows how the store responds internally to these filter requests and efficiently determines the results.

InMemoryUrlMappingStore – Filtering, sorting, and paginating

The InMemoryUrlMappingStore, which already served as a simple storage solution in [Part II], has now been extended by a powerful filter logic. The goal is to process targeted queries directly in memory using the new UrlMappingFilter, including sorting and paging. The code remains easy to understand and test, as no external database is used.

Compared to the original variant, which provided only findAll(), the new implementation has two central extensions:

• find(UrlMappingFilter filter) – returns a filtered, sorted, and paginated list of results.
• count(UrlMappingFilter filter) – determines the number of elements that correspond to a given filter.

An excerpt shows the core of the new logic:

xmlCopy

@Overridepublic List<ShortUrlMapping> find(UrlMappingFilter filter) { Objects.requireNonNull(filter, "filter"); List<ShortUrlMapping> tmp = new ArrayList<>(); for (ShortUrlMapping mapping : store.values()) { if (matches(filter, mapping)) tmp.add(mapping); } Comparator<ShortUrlMapping> cmp = buildComparator(filter); if (cmp != null) { tmp.sort(cmp); if (filter.direction().orElse(Direction.ASC) == Direction.DESC) { Collections.reverse(tmp); } } int from = Math.max(0, filter.offset().orElse(0)); int lim = filter.limit().orElse(Integer.MAX_VALUE); if (from >= tmp.size()) return List.of(); int to = Math.min(tmp.size(), from + Math.max(0, lim)); return tmp.subList(from, to);}

The process is divided into three phases:

1. Filter – matches(filter, mapping) checks substrings (with optional case sensitivity) as well as time periods (createdFrom, createdTo).
2. Sort – the Comparator construct allows sorting by CREATED_AT, SHORT_CODE, ORIGINAL_URL, or EXPIRES_AT.
3. Paging – offset and limit return only the requested subsets.

An example illustrates the effect:

Filter: codePart=“ex-”, sortBy=CREATED_AT, direction=DESC, offset=0, limit=5

Result: The five most recent shortcodes that start with “ex-”.

The count(filter) method , on the other hand, is deliberately trivial:

javaCopy

@Overridepublicintcount(UrlMappingFilterfilter){intc=0;for(ShortUrlMappingm:store.values()){if(matches(filter,m))C++;}returnc;}

This keeps the semantics consistent: count() and find() use the same filter logic, but differ in the return form. This allows pagination information to be reliably calculated.

A key design goal was backwardcompatibility : the original findAll() method remains unchanged. If no filter is passed, find() can fall back on it internally. Thus, older parts of the application continue to work unchanged.

The expansion of the store is an example of how more complex search mechanisms can be implemented using simple Java constructs – type-safe, transparent, and without additional dependencies. The following section shows how this logic is specifically supported and validated by QueryUtils.

QueryUtils – Parsing and Normalising Query Parameters

To enable REST endpoints to respond cleanly and robustly to parameter requests, a small helper class, QueryUtils, has been introduced. It converts raw query strings from HTTP requests into type-safe values. This step relieves the handler classes (ListHandler and ListCountHandler) and at the same time ensures uniform behavior when dealing with user parameters.

The class is located in the package com.svenruppert.urlshortener.api.utils and is purely static. Their purpose is to catch erroneous input, set default values, and convert strings into concrete enums or numbers. This prevents incomplete or incorrect query parameters from causing exceptions or leading to inconsistent states.

An excerpt from the implementation:

xmlCopy

public final class QueryUtils { private QueryUtils() { } public static int parseIntOrDefault(Strings s, int def) { try { return (s == null || s.isBlank()) ? def: Integer.parseInt(s.trim()); } catch (NumberFormatException e) { return def; } } public static int clamp(int v, int lo, int hi) { return Math.max(lo, Math.min(hi, v)); } public static Optional<UrlMappingFilter.SortBy> parseSort(Strings) { if (s == null) return Optional.empty(); return switch (s.toLowerCase(Locale.ROOT)) { case "createdat" -> Optional.of(UrlMappingFilter.SortBy.CREATED_AT); case "shortcode" -> Optional.of(UrlMappingFilter.SortBy.SHORT_CODE); case "originalurl" -> Optional.of(UrlMappingFilter.SortBy.ORIGINAL_URL); case "expiresat" -> Optional.of(UrlMappingFilter.SortBy.EXPIRES_AT); default -> Optional.empty(); }; } public static Optional<UrlMappingFilter.Direction> parseDir(Strings) { if (s == null) return Optional.empty(); return switch (s.toLowerCase(Locale.ROOT)) { case "asc" -> Optional.of(UrlMappingFilter.Direction.ASC); case "desc" -> Optional.of(UrlMappingFilter.Direction.DESC); default -> Optional.empty(); }; }} 

This small utility class does several things at once:

• Resilience: Incorrect or missing parameters never lead to exceptions.
• Consistency: Sorting and direction are interpreted consistently throughout the system.
• Maintainability: Traders no longer have to worry about low-level parsing.

Clamping (clamp(pageSize, 1, 500)) is a crucial security feature: it prevents oversized queries and thus protects both the server and the UI from excessive data volume.

An example illustrates the practical effect:

Input: size=-5 → Result: size=1

Input: size=9999 → Result: size=500

This makes QueryUtils an inconspicuous but essential part of API robustness. Its features follow the same guiding principle that runs through the entire project:explicit typing, clear boundaries, and defensive processing.

Introduction to API Endpoints

Now that the server-side logic has been extended to include filtering, sorting and paging, the REST API itself is now taking centre stage. It serves as the link between the internal functions of the URL shortener and the client’s or UI’s external access. This chapter aims to describe the new and extended endpoints in detail and explain their functions within the overall system.

The focus is not only on the data format of the answers, but also on the semantics of the supported parameters. Both aspects are crucial to make filter queries consistent, comprehensible and efficient. Thanks to a clearly structured API, both existing clients can continue to work, and new components – such as the Vaadin-based administration interface – can access subsets of data in a targeted manner without overloading the system.

Structure of the new endpoints

With the introduction of filtering and paging, the REST API for the URL shortener has also been extended. In addition to the familiar endpoints from [Part II], two new paths are now available that are specifically designed for targeted queries:

javaCopy

GET/list–returnsfilteredandpaginatedresultsGET/list/count–returnsonlythenumberofhits

Both endpoints use the same set of query parameters. While/list returns the actual mappings in structured JSON,/list/count provides a lightweight way to determine the total set for a query. This separation deliberately follows the single-responsibilityprinciple and, at the same time, supports high-performance UIs that load pages and large amounts of data.

A typical call to the new/list endpoint might look like this:

javaCopy

GET/list?code=ex-&url=docs&page=2&size=25&sort=createdAt&dir=desc

The result is a JSON object with the following keys:

jsonCopy

{"mode":"filtered","page":2,"size":25,"total":123,"sort":"createdAt","dir":"desc","count":25,"items":[{"shortCode":"ex-beta","originalUrl":"https://example.org/blog","createdAt":"2025-10-24T12:45:33Z","expiresAt":"","Status":"Active"}]}

The JSON structure follows a clear logic: all metadata for the request is in the first fields, while theitems array contains the actual results. This makes it easy to integrate the structure into UI components such as tables, grids or data providers.

Compatibility and stability

Existing endpoints such as/list/all,/list/active, and/list/expired are fully preserved. They continue to provide unchanged responses, so existing clients don’t need any customisations. The new endpoints are thereforeintegrated additively into the system.

For new clients, especially those using the Vaadin UI presented in [Part III], the introduction of these endpoints provides a basis for reactive data display. Instead of loading all mappings at once, the interface can request only the data sets defined by the current filter.

This API structure lays the foundation for detailing the supported parameters and their meanings in the following subchapters.

Supported Parameters

The new endpoints/list and/list/count accept a set of query parameters that can be used to formulate targeted search and filter queries. All parameters are optional and can be freely combined. Unset fields do not impose any restrictions.

Parameter	Type	Description
code	String	Substring of the short code. Is handled by defaultcase-insensitive ifcodeCase=true is not set.
codeCase	Boolean	Controls whether the search for shortcodes should be case-sensitive.
URL	String	Substring within the original URL. Similar tocode, the case can be controlled via urlCase.
urlCase	Boolean	Case sensitivity switch for URL search.
from	ISO-8601 timestamp	Lower limit of the creation period (inclusive). Also accepts dates without a time component.
To	ISO-8601 timestamp	Upper limit of the creation period (included).
page	Int	1-based page number. Default value:1.
Size	Int	Number of records per page. Values outside therange 1-500 are automatically limited.
sort	String	Sort. Allowed are:createdAt,shortCode,originalUrl,expiresAt.
you	String	Sorting direction:asc ordesc. Default value:asc.

For example, a complete request might look like this:

sqlCopy

GET/list?code=ex-&url=docs&from=2025-10-01T00:00:00Z&to=2025-10-25T23:59:00Z&page=2&size=25&sort=createdAt&dir=desc

The API validates all parameters using theQueryUtils class. Invalid or missing values are replaced with default values to avoid exceptions. This defensive strategy ensures a high level of stability in continuous operation.

Interaction of the parameters

• If neithercode norurl is set, all mappings are taken into account.
• from andto define aninclusive period ; both fields may be set independently of each other.
• page andsize only affect the display of results, not the count in/list/count.
• Combinations ofsort anddir have a consistent effect on both endpoints (/list and/list/count).

A minimalist example of a count-only query is:

javaCopy

GET/list/count?url=example→{"total":12}

The API is designed so that future parameters can be easily added. Thanks to the centralUrlMappingFilter, new fields need only be stored there and taken into account in the builder. This completely preserves the system’s extensibility.

Client-side extensions

After the server-side has been extended with flexible filtering and paging functions, the client-side customisation now follows. This chapter aims to demonstrate how the new data retrieval capabilities have been integrated into the existingURLShortenerClient without altering its structure or semantics.

The focus is ontype-safe communication between the client and the server. Instead of assembling manual query strings or passing loose maps, the new builderUrlMappingListRequest encapsulates all parameters in a clear, object-oriented form. On this basis, filters, sorting and paging can be centrally managed and easily tested.

The chapter highlights the three main aspects of client extension:

1. The new request builder (UrlMappingListRequest), which cleanly encapsulates filter and paging parameters.
2. The advanced methods in the**URLShortenerClient**process these requests and forward them to the new endpoints.
3. The associated test class that secures the interaction between client and server.

Together, these elements serve as the counterpart to the server-side extensions from the chapter “Server-side changes” and provide the basis for an interactive user interface that works specifically with filtered and paginated data.

UrlMappingListRequest – Filter and Paging Request Builder

To enable the client to communicate with the new filter and paging endpoints in a targeted manner, theUrlMappingListRequest class was introduced. It acts as a transportable request object that contains all relevant parameters and, if necessary, translates them into a query string for HTTP communication.

The design follows the principles from [Part II]: no external JSON serialisation, no dependencies on frameworks – instead pure Java logic with a focus on readability and type safety.

An example illustrates the usage:

kotlinCopy

varreq=UrlMappingListRequest.builder().urlPart("docs").page(2).size(25).sort("createdAt").dir("desc").build();

The class then converts this information into a URL query string that can be sent directly to the server:

javaCopy

/list?url=docs&page=2&size=25&sort=createdAt&dir=desc

Internally,UrlMappingListRequest consists of a set of simple fields, such ascodePart,urlPart,from,to,page,size,sort,anddir. The integrated builder ensures that only valid combinations can be formed. Unset values are automatically ignored during serialisation – empty parameters do not appear in the query string.

Two methods are central:

• **toQueryString()** – generates the complete query string including paging and sorting parameters.
• **toQueryStringForCount()** – returns only the filter parameters, without paging or sorting information, for/list/count.

Both variants are based on an internal helper method that checks parameters, sorts them by key, and securely encodes them via URLEncoder:

javaCopy

privatestaticStringtoQuery(Map<String,String>params){returnparams.entrySet().stream().map(e->enc(e.getKey())+"="+enc(e.getValue())).collect(Collectors.joining("&"));}

This structure ensures that all parameters are transmitted cleanly and in accordance with the URL format – a detail particularly essential for strings containing special characters (e.g., in URLs).

Thus,UrlMappingListRequest forms the direct bridge between the client and the API. It does not replace existing calls, but expands the communication options with flexible, type-safe filter queries – in line with the modular system design.

Extension of the URLShortenerClient – Filter and Count

On the client side, theURLShortenerClient has been enhanced with two essential features to target the new server-side endpoints:list(UrlMappingListRequest request) andlistCount(UrlMappingListRequest request). These methods enable dynamic reaction to filter criteria without manual URL assembly.

New methods

The implementation follows the principle of clear separation of responsibilities:list() handles the actual data,listCount() handles the metadata.

xmlCopy

public List<ShortUrlMapping> list(UrlMappingListRequest request) throws IOException { final String json = listAsJson(request); return parseItemsAsMappings(json);}public int listCount(UrlMappingListRequest req) throws IOException { String qs = (req == null) ? "" : req.toQueryStringForCount(); var uri = qs.isEmpty() ? serverBaseAdmin.resolve(PATH_ADMIN_LIST_COUNT) : serverBaseAdmin.resolve(PATH_ADMIN_LIST_COUNT + "?" + qs); var con = (HttpURLConnection) uri.toURL().openConnection(); con.setRequestMethod("GET"); con.setRequestProperty("Accept", "application/json"); int sc = con.getResponseCode(); if (sc != 200) { String err = readAllAsString(con.getErrorStream() != null ? con.getErrorStream() : con.getInputStream()); throw new IOException("Unexpected HTTP " + sc + " for " + uri + " body=" + err); } try (var is = con.getInputStream()) { String body = readAllAsString(is); int i = body.indexOf("\"total\""); if (i< 0)return0;intcolon =body.indexOf(':',i);intend =body.indexOf('}',colon+1);Stringnum =body.substring(colon+1,end).trim();returnInteger.parseInt(num);}finally{con.disconnect();}}

Significance of enlargement

These two methods make the client fully compatible with the advanced server features. The filter logic is completely mapped via theUrlMappingListRequest object, so that the call itself always remains clear:

xmlCopy

var req = UrlMappingListRequest.builder() .codePart("ex-") .urlPart("docs") .page(1) .size(25) .build();List<ShortUrlMapping> results = client.list(req);int total = client.listCount(req);

The client can therefore first query the total amount to calculate pagination, then load only the relevant data pages. This significantly reduces the network load, especially with large amounts of data.

Backward compatibility

All existing methods such aslistAllAsJson() orlistActiveAsJson() are preserved. This keeps the APIbinary and semantically compatible – existing tests and applications continue to work. The new functions fit seamlessly into the existing design and, at the same time, form the basis for a reactive UI that can dynamically evaluate filters and paging.

Client Test Coverage – URLShortenerClientListTest

To secure the new client functions, a separate test class, URLShortenerClientListTest, has been introduced. It checks the interaction between the client and a runningShortenerServer and thus serves as an integration test for the entire filter and paging chain.

The test starts a whole server on a random port and then communicates with the real HTTP endpoint. This ensures that not only the client logic, but also the serialisation, request routing, and server-side filtering work correctly.

An excerpt from the test case:

xmlCopy

@Test@Order(1)void list_all_and_filtered_by_code_and_url_and_date() throws Exception { Arrange var t0 = Instant.now(); ShortUrlMapping m1 = client.createCustomMapping("ex-alpha", "https://example.com/docs"); Thread.sleep(10); ShortUrlMapping m2 = client.createCustomMapping("ex-beta", "https://example.org/blog"); Thread.sleep(10); ShortUrlMapping m3 = client.createMapping("https://docs.example.com/page"); var t1 = Instant.now(); Act – test different filters var reqCode = UrlMappingListRequest.builder().codePart("ex-").build(); List<ShortUrlMapping> byCode = client.list(reqCode); assertTrue(byCode.size() >= 2); var reqUrl = UrlMappingListRequest.builder().urlPart("docs").build(); List<ShortUrlMapping> byUrl = client.list(reqUrl); assertTrue(byUrl.stream().anyMatch(m -> m.originalUrl().contains("docs"))); var reqDate = UrlMappingListRequest.builder().from(t0).to(t1).build(); List<ShortUrlMapping> byDate = client.list(reqDate); assertFalse(byDate.isEmpty());}

The test examines three central use cases:

1. Code filtering – only shortcodes that start with a specific pattern.
2. URL filtering – URLs that contain specific substrings.
3. Time Filtering – Entries created within a specific time window.

Paging and sorting tests

A second test section focuses on pagination and sorting:

xmlCopy

@Test@Order(2)void list_pagination_and_sorting() throws Exception { for (int i = 0; i< 10;i++){client.createMapping("https://example.net/page-"+i);Thread.sleep(2);}varreq =UrlMappingListRequest.builder().page(2).size(5).sort("createdAt").dir("desc").build();List<ShortUrlMapping> page2 = client.list(req); assertEquals(5, page2.size()); for (int i = 1; i< page2.size();i++){assertTrue(!page2.get(i).createdAt().isAfter(page2.get(i-1).createdAt()));}}

This checks that page numbering is working correctly and that the results are returned in the expected order.

Aim of the tests

This test class ensures that the interaction between client and server remains stable, even if parameter combinations change. By using a realHttpServer backend, a realistic environment is simulated, without external dependencies. The result is a high level of trust in the implementation with full coverage of all critical paths.

TheURLShortenerClientListTest class thus concludes the client chapter: It proves that the system as a whole – from filter definition to HTTP communication to result processing – functions consistently.

In the next part, we will then look at the integration into the Vaadin UI.

Cheers Sven

]]>Design PatternJavaServersidehttps://svenruppert.com/posts/introduction-to-the-url-shortener-advent-calendar-2025/Mon, 01 Dec 2025 10:22:35 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/introduction-to-the-url-shortener-advent-calendar-2025/December 2025 is all about a project that has grown steadily in recent months: the Java-based URL Shortener, an open-source project implemented entirely with Core Java, Jetty, and Vaadin Flow. The Advent calendar accompanies users every day with a new feature, a technical deep dive, or an architectural improvement – from the basic data structure and REST handlers to UI components and security aspects.<![CDATA[

December 2025 is all about a project that has grown steadily in recent months:the Java-based URL Shortener, an open-source project implemented entirely with Core Java, Jetty, and Vaadin Flow. The Advent calendar accompanies users every day with a new feature, a technical deep dive, or an architectural improvement – from the basic data structure and REST handlers to UI components and security aspects.

1. Motivation: Why your own URL shortener?
2. What can users expect in the Advent calendar?
3. Reasons for the choice of technology
   1. Core Java (Java 24)
   2. Jetty as an embedded web server
   3. Vaadin Flow for the UI
   4. EclipseStore as a persistence solution
4. Reference to previously published articles
5. Why, as an Advent calendar?

Motivation: Why your own URL shortener?

The entire project is developed openly on GitHub. There, users will not only find the current state of the code, but also all historical development steps, patches, branches, and PRs discussed in the context of the Advent calendar. The source code can be viewed athttps://github.com/svenruppert/url-shortener.

Commercial URL shorteners are practical – but often non-transparent. They collect usage data, offer proprietary APIs, or block essential functions behind paywalls. The goal of this project is therefore a completely self-powered, transparent, and secure shortener suitable for internal networks, development teams, companies, and hobby projects alike.

The project pursues three central basic principles: transparency, in which every decision is documented in the code in a comprehensible way; didactics, as the project is not only intended to be a tool, but above all a learning platform; and extensibility, because each feature is designed to be modular and can therefore be easily adapted or expanded.

What can users expect in the Advent calendar?

The technical journey through Advent is visible both in terms of content and directly in the repository. Each day is created as its own Git branch, starting withfeature/advent-2025-day-01 and continuing until the final Christmas door. This allows the project’s development to be traced in small, clearly defined steps – including all refactorings, architectural adjustments, and new functions. In the same structure, the Advent Calendar presents a precisely outlined feature every day, which is reproduced using the associated patch or PR. Supplementary code excerpts from the core and UI modules, architectural explanations and references to typical anti-patterns deepen the understanding. Each door is rounded off with suitable visualisations and a compact summary, so that the project’s development becomes transparent and tangible step by step.

Reasons for the choice of technology

For the URL shortener, technologies were deliberately chosen that are bothpractical anddidactically valuable. Every decision supports the project’s goal: to create a modern yet easy-to-understand system that requires no unnecessary dependencies.

Core Java (Java 24)

The project’s basis is pure Core Java. The reasons for this are:

The use of pure core Java enables maximum transparency: Without any framework magic, users can see exactly how data models, handlers, error handling or serialisation are structured. By deliberately avoiding additional libraries, the attack surface is reduced, and the risk of supply chain problems is minimised. At the same time, this approach makes it very clear to developers how robust and comprehensive the JDK already is today: Many tasks that used to require external dependencies can now be easily solved with onboard tools.

Jetty as an embedded web server

A deliberate alternative to Spring Boot or Jakarta EE:
Jetty is characterised by its lightweight design and rapid launch time, which are especially beneficial in development scenarios. At the same time, Jetty offers complete control over routing and servlets, so that HTTP mechanics can be precisely demonstrated and implemented in a targeted manner. Thanks to its modular structure, Jetty is ideal for small, well-defined microservices while remaining a stable, proven technology whose long production history speaks for high reliability.

Jetty offers just the right balance between simplicity and technical relevance for this open source project.

Vaadin Flow for the UI

The project uses Vaadin Flow to implement an entirely server-side, Java-focused UI that does not require an additional JavaScript or TypeScript stack, making it particularly suitable for developers who want to focus entirely on Java. Instead of relying on a complex front-end ecosystem, Vaadin enables end-to-end development in a language, significantly reducing the cognitive load and flattening the learning curve. The component-based architecture enables a productive and structured way of working, in which user interfaces can be clearly modelled and reused. At the same time, server-side rendering eliminates the need for direct REST calls from the browser, increasing security and reducing attack surfaces. Despite this technical simplicity, Vaadin offers modern styling and a user experience that is reminiscent of the professionalism of enterprise applications. This makes the framework ideal for internal tools, administrative interfaces and corporate projects where security, robustness and long-term maintainability are paramount.

EclipseStore as a persistence solution

EclipseStore replaces classic databases with an object-oriented persistence approach that deliberately does not require ORM layers, entities, table models or complex abstractions. Instead of having to convert data into a relational structure, it remains in its natural form: Java records, lists and other object structures are directly persisted. This eliminates the usual mapping logic, significantly reducing the complexity of the persistence layer and making the architecture more straightforward to understand.

EclipseStore shows its strength especially in small, focused services. Persistence fits seamlessly into the domain model without forcing developers to adapt their data structures to a relational mindset. This direct approach not only leads to more elegant modelling but also to excellent performance. Since EclipseStore manages in-memory-compliant object graphs, data access is high-speed and does not require any additional caching mechanisms or complex optimisations. The result is a lightweight yet powerful persistence system ideal for a modern, compact Java service.

Didactically, EclipseStore provides valuable insight into how persistent data models can operate when not constrained by relational database constraints. Developers gain a deeper understanding of what object lifecycles, data flows, and modelling decisions look like in an environment that is designed entirely from the Java world. This makes EclipseStore particularly suitable for this URL shortener.

Reference to previously published articles

Before we get into the Advent calendar, it’s worth taking a look at the previous publications, which already present the basics, architecture and the first UI concepts of the project. These articles form the foundation on which the content of the Advent calendar is built:

• Part I – Short links, clear architecture : A detailed introduction to the motivation, architecture and data model of the project. The article shows how to develop a URL shortener entirely in Core Java, along with the design principles behind it. (https://svenruppert.com/2025/06/10/short-links-clear-architecture-a-url-shortener-in-core-java/ )
• Part II – Deepening the Server Components : This article explores REST handlers, validation, security considerations, and the interaction between the core module and the server layer. (https://svenruppert.com/2025/06/20/part-ii-urlshortener-first-implementation/)
• Part III – The Web UI with Vaadin Flow: This shows how the user interface is structured, how lists, forms, dialogues and validations work, and how Vaadin Flow enables a modern UI without an additional JavaScript stack. (https://svenruppert.com/2025/08/15/part-iii-webui-with-vaadin-flow-for-the-url-shortener/)

These items offer a first technical introduction and are perfect as preparation for the individual doors of the Advent calendar.

Why, as an Advent calendar?

A project like a URL shortener doesn’t happen in a weekend – it consists of many small steps, refactorings, decisions and improvements. An Advent calendar offers the perfect format for this:short, daily, understandable bites that come together to form an overall understanding.

It is, at the same time, a review of the project’s journey and a motivating start to the new year: a project that users can run, expand, change, or completely rethink.

]]>EclipseStoreJavaVaadinhttps://svenruppert.com/posts/real-time-in-focus-server-sent-events-in-core-java-without-frameworks/Fri, 05 Sep 2025 08:18:47 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/real-time-in-focus-server-sent-events-in-core-java-without-frameworks/Chapter 1 – Introduction 1.1 Motivation: Real-time communication without polling In modern applications, it is often necessary to provide new information to the client as quickly as possible. Classic polling , i.e. regularly querying a REST endpoint, is inefficient: it generates unnecessary network traffic and puts a load on both server and client, as requests continue even when there is no new data.<![CDATA[

Chapter 1 – Introduction

1.1 Motivation: Real-time communication without polling

In modern applications, it is often necessary to provide new information to the client as quickly as possible. Classicpolling , i.e. regularly querying a REST endpoint, is inefficient: it generates unnecessary network traffic and puts a load on both server and client, as requests continue even when there is no new data.

Server-sent events (SSE) offer a resource-saving alternative here. Instead of the client constantly sending new requests, it maintains an open HTTP connection over which the server can send messages at any time. This creates a continuous stream of data with minimal overhead.

1. Chapter 1 – Introduction
   1. 1.1 Motivation: Real-time communication without polling
   2. 1.2 Differentiation from WebSockets and Long Polling
   3. 1.3 Objective of the article
2. Chapter 2 – The SSE Protocol
   1. 2.1 MIME type text/event-stream
   2. 2.2 Message structure: event:, data:, id: and comments
   3. 2.3 Automatic Reconnect and Event IDs
   4. 2.4 Limitations: UTF-8, Unidirectionality, Proxy Behaviour
3. Chapter 3 – Minimal REST/SSE Server in Core Java
   1. 3.1 Implementation with HttpServer
   2. 3.2 Sending Events (Text, Comments, IDs)
   3. 3.3 Keep-Alive and Connection Stability
   4. 3.4 Step-by-step implementation – complete minimal example
4. Chapter 4 – Minimal SSE Client in Core Java
   1. 4.1 Implementation with HttpClient
   2. 4.2 Interpreting Events
   3. 4.3 Behaviour in case of aborting and reconnect
5. Chapter 5 – Testing and Debugging SSE
   1. 5.1 Using curl and browser EventSource
   2. 5.2 Checking with Java Client
   3. 5.3 Monitoring and logging
6. Chapter 6 – Application Scenarios and Limitations

1.2 Differentiation from WebSockets and Long Polling

In the context of current communication paradigms, three basic approaches can be distinguished. In so-calledlong polling , the client initiates a request that the server keeps open as long as possible until new information is available. After this has been transmitted, the connection is closed, whereupon the client immediately opens a new request. Although this method appears to be more efficient than classic polling, it remains resource-intensive due to the frequent reinitialization of the connections.

In contrast,WebSockets establish a full-duplex communication relationship. Here, both client and server can exchange messages continuously and bidirectionally. This model offers high flexibility and performance, but it also introduces increased complexity in implementation and operation, often resulting in oversizing when only server-side push messages are needed.

Server-sent events (SSE)act as an intermediary approach. They establish a standardised, unidirectional data stream from the server to the client via a persistent HTTP connection. The native support of current browsers and the comparatively low implementation effort on the server side make SSE a practical option, especially in scenarios where simplicity and resource conservation are paramount.

Sourcecode is on GitHub
https://github.com/Java-Publications/Blog---Java---Server-Sent-Events-SSE-in-Core-Java-Basics-and-Implementation

1.3 Objective of the article

This article provides a basic introduction toServer-Sent Events (SSE) using Core Java. The focus is on consistent implementation based on theJava Development Kit (JDK), without recourse to external frameworks. The aim is to analyse the specifics of the protocol as well as the minimal implementations on both the server side and the client side in a systematic form and to present them in a comprehensible manner.

In the course of the study, it is illustrated how an elementary REST/SSE server can be constructed in Java, how a client can receive and process event streams using the Java HttpClient, which typical difficulties arise in the context of testing and debugging, and in which application scenarios the use of SSE offers significant added value.

Finally, the position of SSE in the area of tension between polling and WebSocket-based methods is critically classified and the transition to the following, practice-oriented implementation chapters is prepared.

Chapter 2 – The SSE Protocol

2.1 MIME typetext/event-stream

The foundation of server-sent events is the dedicated MIME typetext/event-stream. As soon as a server declares this Content-Type in the HTTP header, it signals to the client that the subsequent data is to be interpreted as a continuous event stream. In contrast to conventional textual response formats, which are closed after complete transmission, SSE deliberately keeps the connection open persistently. This semantic definition creates the basis for a resource-saving push model that can be integrated into existing HTTP infrastructures without proprietary extensions.

2.2 Message structure:event: ,data: ,id: and comments

An SSE data stream is organised on a row-based basis. Each message can consist of multiple fields preceded by a keyword and a colon. The most important fields are:

• event: defines the type of event that the client can handle specifically.
• data: contains the actual payload. Consecutive data lines summarise multi-linedata fields.
• id: represents an event ID that is used for resumption after disconnections.
• Comments: start with a colon and are used both for semantic documentation and to maintain keep-alive signals.

The delimitation of individual events is done by a blank line (double line breaks), which completes the semantic unit of a message.

2.3 Automatic Reconnect and Event IDs

A significant advantage of SSE over simpler push mechanisms is the integrated reconnect behaviour. If a connection is unexpectedly interrupted, the client automatically initiates a new connection. Using theLast Event ID HTTP header fields or explicit ID assignments in the data stream, it is possible to continue at the exact point where the data stream was interrupted. This principle reduces data loss and facilitates the implementation of robust, state-preserving communication patterns.

2.4 Limitations: UTF-8, Unidirectionality, Proxy Behaviour

Despite its simplicity, the protocol has inherent limitations. On the one hand, the character encoding is strictly set to UTF-8, which means that binary payloads cannot be transmitted directly. In addition, the direction of communication is limited to server-side messages; a bidirectional interaction requires recourse to alternative methods such as WebSockets. Finally, the behaviour of intermediary network nodes – such as proxies or load balancers – can affect the longevity and stability of open HTTP connections, which deserves special attention in production environments.

Chapter 3 – Minimal REST/SSE Server in Core Java

3.1 Implementation withHttpServer

An elementary SSE server can be implemented in Java using the classes included in the JDK. Of particular note is the com.sun.net.httpserver.HttpServer class, which allows HTTP endpoints to be provided without external dependencies. Such a server can be created with a few lines of code, bound to a port number and extended by handlers that process incoming requests.

A minimal basic framework looks like this:

javaCopy

HttpServerserver=HttpServer.create(newInetSocketAddress(8080),0);server.createContext("/events",exchange->{   exchange.getResponseHeaders().add("Content-Type","text/event-stream");   exchange.sendResponseHeaders(200,0);   OutputStreamos=exchange.getResponseBody();   //This is where the events will be written later});server.start();

It is crucial to set the content type totext/event-stream and not to close the HTTP connection immediately. This creates the basis for continuously transmitting event data to the client.

3.2 Sending Events (Text, Comments, IDs)

The server-side logic for transmitting messages follows the formats defined in the protocol. A typical message can contain both adata: field with payload and optional metadata. Comments are beneficial for realising keep-alive signals.

A simple example of how to transmit messages:

javaCopy

PrintWriterwriter=newPrintWriter(os,true);writer.println("id: 1");writer.println("event: greeting");writer.println("data: Hello client!");writer.println();Endmessagewriter.flush();//Comment as a heartbeatwriter.println(": keep-alive");writer.println();writer.flush();

Here, a message is transmitted with ID, event type, and data. The double line break concludes the semantic unit. The comment is to maintain the connection.

3.3 Keep-Alive and Connection Stability

A critical aspect of running SSE servers is ensuring connection stability. HTTP connections can be terminated by inactivity or by restrictive network infrastructures. To avoid this, it is common practice to periodically send comments or blank messages that are ignored by the client but registered as activity by the network. This technique, often referred to as heartbeat or keep-alive, helps keep the connection stable for extended periods of time.

A robust implementation should also ensure that there are no resource leaks when the connection is lost. In particular, this means that OutputStreams must be reliably closed and background threads must be terminated in a controlled manner.

This step-by-step implementation creates a functional SSE server that meets the fundamental protocol requirements. In the following chapters, this is built on by developing a corresponding client and examining the robustness of the communication.

3.4 Step-by-step implementation – complete minimal example

In the following, an executable minimal server is set up in clearly defined steps. The implementation uses only the JDK (no external dependencies) and relies on com.sun.net.httpserver.HttpServer.

**Create and start the server

javaCopy

packagecom.svenruppert.sse;importcom.sun.net.httpserver.HttpServer;importcom.svenruppert.dependencies.core.logger.HasLogger;importjava.io.*;importjava.net.InetSocketAddress;import staticjava.nio.charset.StandardCharsets.UTF_8;publicclassSseServer   implementsHasLogger{ protectedstaticfinalStringPATH_SSE="/sse"; privatevolatilebooleansendMessages=false; privatevolatilebooleanshutdownMessage=false; // Reference so we can properly stop the server from the CLI privateHttpServerserver; publicvoidstart()     throwsException{   intport=8080;   this.server=HttpServer.create(newInetSocketAddress(port),0);   // Background thread for CLI control (start | stop | shutdown)   Threadcli=newThread(()->{     try(BufferedReaderconsole=newBufferedReader(newInputStreamReader(System.in))){       Stringline;       while((line=console.readLine())!=null){         Stringcmd=line.trim().toLowerCase();         switch(cmd){           case"start"->cmdStart();           case"stop"->cmdStop();           case"shutdown"->cmdShutdown();           default->logger().info("Unknown command: {} (allowed: start | stop | shutdown)",cmd);         }       }     }catch(IOExceptionioe){       logger().warn("CLI control terminated: {}",ioe.getMessage());     }   },"cli-control");   cli.setDaemon(true);   cli.start();   server.createContext(PATH_SSE,exchange->{     if(!"GET".equalsIgnoreCase(exchange.getRequestMethod())){       exchange.sendResponseHeaders(405,-1);       return;     }     exchange.getResponseHeaders().add("Content-Type","text/event-stream; charset=utf-8");     exchange.getResponseHeaders().add("Cache-Control","no-cache");     exchange.getResponseHeaders().add("Connection","keep-alive");     exchange.sendResponseHeaders(200,0);     try(OutputStreamos=exchange.getResponseBody();          OutputStreamWriterosw=newOutputStreamWriter(os,UTF_8);          BufferedWriterbw=newBufferedWriter(osw);          PrintWriterout=newPrintWriter(bw,true)){       longid=0;       while(!shutdownMessage){         if(sendMessages){           id++;           Stringdata="tick @"+System.currentTimeMillis();           writeEvent(out,"tick",data,Long.toString(id));         }else{           heartbeat(out);         }         Thread.sleep(1000);       }       // Optional: send farewell event before shutting down       writeEvent(out,"shutdown","Server is shutting down",Long.toString(++id));     }catch(IOException|InterruptedExceptionioe){       logger().warn("SSE client disconnected: {}",ioe.getMessage());     }finally{       logger().info("SSE stream closed");     }   });   server.start();   logger().info("SSE server running at http://localhost:{}/sse",port);   Runtime.getRuntime().addShutdownHook(newThread(()->{     logger().info("Stopping server …");     if(server!=null){       server.stop(0);     }   })); } privatevoidcmdShutdown(){   logger().info("Shutdown command received – 'shutdown' event will be sent …");   shutdownMessage=true;// signals all active handlers to send a shutdown event   try{     // Grace period so handlers can still execute writeEvent(..., "shutdown", ...) + flush     Thread.sleep(1200);   }catch(InterruptedExceptionie){     Thread.currentThread().interrupt();   }   try{     if(server!=null){       server.stop(0);// stop HTTP server afterwards     }   }catch(Exceptione){     logger().warn("Error while stopping: {}",e.getMessage());   }   logger().info("Application is shutting down.");   System.exit(0); } privatevoidcmdStop(){   sendMessages=false;   logger().info("SSE message sending stopped via CLI"); } privatevoidcmdStart(){   sendMessages=true;   logger().info("SSE message sending started via CLI"); } // Helper function for sending an event in SSE format privatevoidwriteEvent(PrintWriterout,Stringevent,Stringdata,Stringid){   if(event!=null&&!event.isEmpty()){     out.printf("event: %s%n",event);   }   if(id!=null&&!id.isEmpty()){     out.printf("id: %s%n",id);   }   if(data!=null){     // Correctly output multiline data     for(Stringline:data.split("\\R",-1)){       out.printf("data: %s%n",line);     }   }   out.print("\n");// terminate message with empty line   out.flush(); } // Comment-based heartbeat, ignored by client – keeps connection alive privatevoidheartbeat(PrintWriterout){   out.print(": keep-alive\n\n");   out.flush(); }}

**Quick function test

• Start: java Application (or compile/execute via javac/java).
• Check with a small Java program:

javaCopy

importjava.net.URI;importjava.net.http.HttpClient;importjava.net.http.HttpRequest;importjava.net.http.HttpResponse;import staticjava.net.http.HttpClient.newHttpClient;publicclassSseTestClient{ publicstaticvoidmain(String[]args)     throwsException{   // Create a new HTTP/2 client   HttpClientclient=newHttpClient();   // Build GET request to connect to the SSE endpoint   HttpRequestrequest=HttpRequest.newBuilder()       .uri(URI.create("http://localhost:8080/sse"))       .GET(       .build();   // Asynchronously send request and consume the response stream line by line   client.sendAsync(request,HttpResponse.BodyHandlers.ofLines())       .thenAccept(response->           response.body().forEach(System.out::println))       .join(); }}

This simple program opens a connection to the server and outputs the received SSE messages line by line to the console. Continuous event:/id:/data: blocks and : keep-alive comments are expected.

Chapter 4 – Minimal SSE Client in Core Java

4.1 Implementation withHttpClient

On the client side, the JDK has offered a powerful API since Java 11 with java.net.http.HttpClient to process HTTP connections asynchronously. Because SSE is based on a persistent GET request, the client can continuously receive incoming data via HttpClient. In contrast to classic REST requests, which are completed after the body has been entirely accepted, the connection remains open and delivers lines intext/event-stream format.

We encapsulate the implementation in a dedicated class SseClient, which contains no static methods. The client is then started via a separate class SseClientApp.

xmlCopy

package com.svenruppert.sse.client;import com.svenruppert.dependencies.core.logger.HasLogger;import java.io.IOException;import java.net.URI;import java.net.http.HttpClient;import java.net.http.HttpRequest;import java.net.http.HttpResponse;import java.time.Duration;import java.util.function.Consumer;import java.util.stream.Stream;/** * Minimal, instance-based SSE client built on java.net.http.HttpClient. * - manages Last-Event-ID * - implements a reconnect loop * - parses text/event-stream */public final class SseClient    implements HasLogger, AutoCloseable {  private final HttpClient http;  private final URI uri;  private final Duration reconnectDelay = Duration.ofSeconds(2);  private volatile boolean running = true;  private volatile String lastEventId = null;  public SseClient(URI uri) {    this.uri = uri;    this.http = HttpClient.newBuilder()        .connectTimeout(Duration.ofSeconds(5))        .build();  }  /**   * Starts the streaming loop and invokes the callback for each complete event.   */  public void run(Consumer<SseEvent> onEvent) {    while (running) {      try {        HttpRequest.Builder b = HttpRequest.newBuilder()            .uri(uri)            .timeout(Duration.ofMinutes(10))            .GET();        if (lastEventId != null) {          b.setHeader("Last-Event-ID", lastEventId);        }        HttpRequest req = b.build();        // synchronous streaming so we can parse line by line        HttpResponse<Stream<String>> resp =            http.send(req, HttpResponse.BodyHandlers.ofLines());        if (resp.statusCode() != 200) {          logger().warn("Unexpected status {} from {}", resp.statusCode(), uri);          sleep(reconnectDelay);          continue;        }        try {          parseStream(resp.body(), onEvent);        } catch (java.io.UncheckedIOException uioe) {          // Common case on server shutdown: stream is closed → reconnect gracefully          var cause = uioe.getCause();          logger().info("Stream closed ({}). Reconnecting …",                        cause != null                            ? cause.getClass().getSimpleName()                            : uioe.getClass().getSimpleName());        } catch (RuntimeException re) {          logger().warn("Unexpected runtime exception in parser: {}", re.getMessage());        }      } catch (IOException | InterruptedException e) {        if (!running) break; // terminated normally        logger().warn("Connection interrupted ({}). Reconnecting in {}s …",                      e.getClass().getSimpleName(), reconnectDelay.toSeconds());        sleep(reconnectDelay);      }    }  }  private void parseStream(Stream<String> lines, Consumer<SseEvent> onEvent) {    String event = null, id = null;    StringBuilder data = null;    try {      for (String line : (Iterable<String>) lines::iterator) {        if (line.isEmpty()) {          // finish current message          if (data != null) {            String payload = data.toString();            SseEvent ev = new SseEvent(event, payload, id);            if (id != null) lastEventId = id; // remember progress            // shutdown signal from server → client terminates gracefully            if ("shutdown".equalsIgnoreCase(ev.event())) {              try {                onEvent.accept(ev);              } catch (RuntimeException ex) {                logger().warn("Event callback threw exception: {}", ex.getMessage());              }              this.running = false;              break; // leave parsing loop            }            try {              onEvent.accept(ev);            } catch (RuntimeException ex) {              logger().warn("Event callback threw exception: {}", ex.getMessage());            }          }          event = null;          id = null;          data = null;          continue;        }        if (line.startsWith(":")) {          // comment/heartbeat — silently ignore (avoid log spam)          continue;        }        int idx = line.indexOf(':');        String field = (idx >= 0 ? line.substring(0, idx) : line).trim();        String value = (idx >= 0 ? line.substring(idx + 1) : "");        if (value.startsWith(" ")) value = value.substring(1); // drop optional leading space        switch (field) {          case "event" -> event = value;          case "id" -> id = value;          case "data" -> {            if (data == null) data = new StringBuilder();            if (!data.isEmpty()) data.append(' ');            data.append(value);          }          default -> { /* unknown field ignored */ }        }      }    } catch (java.io.UncheckedIOException uioe) {      // Typically EOF/closed when server stops → do not throw further      var cause = uioe.getCause();      logger().info("Input stream ended ({}).",                    cause != null                        ? cause.getClass().getSimpleName()                        : uioe.getClass().getSimpleName());    }  }  private void sleep(Duration d) {    try {      Thread.sleep(d.toMillis());    } catch (InterruptedException ignored) {      Thread.currentThread().interrupt();    }  }  @Override  public void close() { running = false; }}package com.svenruppert.sse.client;import com.svenruppert.dependencies.core.logger.HasLogger;import java.net.URI;public final class SseClientApp implements HasLogger {  public static void main(String[] args) {    String url = (args.length > 0 ? args[0] : "http://localhost:8080/sse");    URI uri = URI.create(url);    try (SseClient client = new SseClient(uri)) {      Runtime.getRuntime().addShutdownHook(new Thread(client::close));      client.run(ev -> {        // You could branch based on ev.event(); for now we just log        if (ev.id() != null) {          System.out.printf("[%s] id=%s data=%s%n", ev.event(), ev.id(), ev.data());        } else {          System.out.printf("[%s] data=%s%n", ev.event(), ev.data());        }      });    }  }}

4.2 Interpreting Events

The SseClient has a parseEvents(Stream) method that parses incoming rows for event:, data:, and id: fields and returns them as SseEvent objects. This allows the received data to be structured and processed in a targeted manner.

4.3 Behaviour in case of aborting and reconnect

The infinite loop mechanism implemented in the start() method ensures that the client automatically establishes a new connection in the event of an abort. A short break is taken between reconnects to avoid endless loops in the event of permanent connection problems.

With this architecture, the client is clearly structured: SseClient encapsulates the logic, while SseClientApp is the starting point for execution. This means that the implementation remains modular, testable and platform-independent.

Hints

• SseClient has no static methods; it is controlled by instance state (running, lastEventId).
• SseClientApp is the entry point. The URL can be passed as an argument; Standard is http://localhost:8080/sse.
• Reconnect takes place automatically with the last received Last Event ID.
• The callback interface (Consumer) enables flexible further processing (logging, UI updates, persistence, etc.).

Chapter 5 – Testing and Debugging SSE

5.1 Using curl and browser EventSource

To check SSE endpoints, developers often turn tocurl because it’s available on almost all platforms. With the -N option (no buffering) the messages can be made directly visible:

curl -N http://localhost:8080/sse

The received events appear in the terminal in their raw form, including event:, data: and id: fields. An alternative is to use the JavaScript API EventSource in the browser:

const source = new EventSource(“http://localhost:8080/sse”);

source.onmessage = e = > console.log(e.data);

This allows the behavior of the server to be tracked directly in the browser.

5.2 Checking with Java Client

Since this article deliberately emphasizes platform independence, a separate Java client is a good idea instead of external tools (see chapter 4). This can be used both as a permanent listener and specifically for test cases, for example by consuming only a certain number of events and then terminating them.

Example: A test run that collects exactly five events and then ends:

xmlCopy

SseClient client = new SseClient(URI.create("http://localhost:8080/sse"));List<SseEvent> events = new ArrayList<>();client.run(ev -> {  events.add(ev);  if (events.size() >= 5) {    client.close();  }});

This allows functional and integration tests to be carried out directly in the Java environment without dependence on curl or telnet .

5.3 Monitoring and logging

Precise monitoring of SSE connections is indispensable for the operation of productive systems. Monitoring is the systematic recording of key figures such as the number of active clients or the average duration of a connection. Equally important is end-to-end error tracking, which documents occurring IO errors, timeouts and aborted transmissions and thus forms the basis for rapid troubleshooting. In addition, continuous monitoring of the heartbeats ensures that comments or keep-alive signals are sent at regular intervals, keeping the connections stable.

Effective logging supports this monitoring and should be done on both the server and client sides. This makes it possible to isolate network problems or interference caused by proxies more quickly. It is beneficial to record unique identifiers for connections and individual events in the log to make communication paths transparent and comprehensible even in complex scenarios.

These tools can be used to reliably track, reproduce and monitor the behaviour of SSE implementations in production environments.

Chapter 6 – Application Scenarios and Limitations

Server-sent events (SSE) find their strength in fields of application where continuous, unidirectional data transmission from the server to the client is required. It is particularly obvious to use it in the field of notification systems. Here, events such as the arrival of new messages, status changes in a workflow or system warnings can be transmitted directly to users without them having to make repeated requests actively. Another typical scenario is monitoring solutions, in which measured values or system metrics are visualised in real time. SSE also offers a resource-saving option for continuous data delivery in telemetry, for example, in the transmission of sensor data from distributed systems.

At the same time, it is essential to consider the limitations of the procedure. Since SSE only transmits UTF-8 encoded text data, it is not possible to transmit binary content directly. Although binary data can be encoded in text form (e.g. using Base64), this is at the expense of efficiency. SSE is also limited to one-way communication: the server can send data to the client, but not vice versa. For scenarios that require bidirectional exchange, WebSockets or similar technologies are more suitable. Network Restrictive problem infrastructures represent another problem area. Proxies or firewalls can block or disconnect long-term open HTTP connections after a certain amount of time, necessitating an additional reconnect strategy.

Safety aspects should also not be neglected when using SSE. Since the connection is maintained via the HTTP protocol, particular attention must be paid to transport encryption using HTTPS. In addition, mechanisms should be implemented to prevent misuse by unauthorised clients, such as authentication and access control to the SSE endpoint. After all, resource conservation is a central issue: While SSE works much more efficiently than polling, the simultaneous supply of a massive number of clients can lead to a high load on server resources. Scaling strategies such as load balancing or splitting clients across multiple SSE endpoints can help here.

In addition, it makes sense to name the functions provided in the specification in detail. The message structure provides several possible fields: event: determines the type of the message and allows differentiated event handlers in the client; data: contains the actual message body and can be continued across multiple lines, with the client automatically combining the contents into a data block; id: allows each message to be uniquely labeled so that the Last-Event-ID header can be seamlessly continued when the connection is re-established; retry: signals to the client after which time an automatic connection should be established in case the connection is interrupted; finally, comments can be inserted via a leading colon: which are ignored by the client but used for keep-alive mechanisms. These fields together form the normative framework that every compliant SSE server should meet.

Overall, SSE is suitable for a wide range of real-time scenarios, as long as the inherent limitations are taken into account and compensated for by appropriate architectural decisions. The technology offers an elegant solution for cases where simplicity, standards compliance, and stable, server-side push communication are paramount.

]]>JavaServersidehttps://svenruppert.com/posts/core-java-flow-processor/Thu, 04 Sep 2025 11:27:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/core-java-flow-processor/Reactive streams address a fundamental problem of modern systems: Producers (sensors, services, user events) deliver data at an unpredictable rate , while consumers (persistence, UI, analytics) can only process data at a limited speed. Without a flow control model , backlogs, storage pressure, and ultimately outages occur.<![CDATA[

Reactive streams address a fundamental problem of modern systems:Producers (sensors, services, user events) deliver data atan unpredictable rate , whileconsumers (persistence, UI, analytics) can only process data at a limited speed. Without aflow control model , backlogs, storage pressure, and ultimately outages occur.

With Java, java.util.concurrent.Flow provides a minimalist API that standardises this problem:Publisher → Processor → Subscriber, includingbackpressure. Flow.Processor<I,O> is thehinge between upstream and downstream:process, transform, buffer, throttle , and at the same time correctly respect the demand.

Abstract: Reactive Streams =asynchronous push model with backpressure. Java Streams =synchronous PullModel without backpressure.

1. 1.1 Why Reactive Streams?
   1. Where does that fit in the JDK?
   2. 1.2 Push vs. Pull: Reactive Streams vs. Java Streams
   3. 1.3 Typical Use Cases
2. 2. Overview: java.util.concurrent.Flow
   1. 2.1 Life cycle of the signals
   2. 2.2 Backpressure basic principle
3. 3. Focus on Flow.Processor<I,O>
   1. 3.1 Contract & Responsibilities
   2. Type Variables & Invariants
   3. Chain position: between upstream & downstream
4. 4. Practical introduction with the JDK
   1. 4.1 SubmissionPublisher in a nutshell
   2. 4.2 Simple pipeline without its own processor
   3. 4.3 Limitations of SubmissionPublisher
5. 5. Sample Processor with Code Examples
   1. 5.1 MapProcessor<I,O> Transformation
   2. 5.2 FilterProcessor – Filtering Data
   3. 5.3 BatchingProcessor – Collect and Share
   4. 5.4 ThrottleProcessor Throttling
6. 6. Backpressure in practice
   1. 6.1 Strategies for dealing with backpressure
   2. 6.2 Demand Policy: Batch vs. Single Retrieval
   3. 6.3 Monitoring, Tuning and Capacity Planning
7. Concurrency & Execution Context
   1. 7.1 Executor Strategies
   2. 7.2 Reentrancy traps and serialization of signals
   3. 7.3 Virtual Threads (Loom) vs. Reactive
8. Error & Completion Scenarios
   1. 8.1 Semantics of onError and onComplete
   2. 8.2 Restarts and Retry Mechanisms
   3. 8.3 Idem potency and exactly-once processing
9. Interoperability & Integration
   1. 9.1 Integration with Reactive Frameworks
   2. 9.2 Use in classic applications
   3. 9.3 Bridge to Java Streams
10. Result

1.1 Why Reactive Streams?

• Controlled load (backpressure): Consumers actively signal how many items they can process (request(s)) by requesting the exact number of items that they can currently process safely via their subscription. The publisher may only deliver as many items as previously requested, maintaining a controlled processing speed, ensuring buffers do not overflow, and keeping memory consumption predictable. The demand can be adjusted dynamically (e.g. batch request(32) or item-wise request(1)). In case of overload, the consumer can pause or cancel cleanly by callingcancel().
• Asynchrony & Decoupling: Data flows can cross thread and service boundaries. This means that a publisher is operated in its own thread or via an executor, for example, while the subscriber processes in a different context. In between, there can be queues or network boundaries, for example, when events are sent from one microservice to another. Loose coupling allows systems to scale and distribute without processing being tightly tied to a single execution context.
• Infinite/long flows: Telemetry, logs, and EventStreams are often continuous, potentially infinite data sources. A single pull, as is standard in classic Java streams, is insufficient here because the data is not available at a fixed time, but rather constantly accumulates. Instead, you need a permanent subscription that continuously informs the consumer about new events. This is the only way to process continuous data streams without the consumer having to poll or repeatedly start new streams actively.
• Error semantics & completion: Consistent signals such as onError and onComplete ensure that each data pipeline receives a well-defined closure or transparent error handling. Instead of unpredictable exceptions that occur somewhere in the code, there are standardised callback methods that clearly mark the lifetime of a stream. This enables the reliable distinction between a data stream that has been terminated regularly and one that has encountered an error, allowing downstream components to react accordingly, such as through cleanup, retry mechanisms, or logging. This predictability is crucial for robust pipelines that are to run stably over more extended periods of time.
• Composability: Composability plays a significant role: A processor can be used as a transformation, for example, by modifying incoming data according to specific rules – similar to a map or filter operation in Java streams. In addition, control operators can be implemented, such as a throttle that limits the rate or a debounce, which only passes on the last signal of a short period of time. Finally, a processor also enables the construction of more complex topologies, such as FanIn, where several sources are merged, or FanOut, where a data stream is distributed among multiple subscribers. This allows entire processing pipelines to be assembled in a modular manner without the individual components having to know their internal logic.
• **Operation & Security: **Backpressure is not only used for load control, but also acts as effective protection against denial‑of service attacks. A system that writes data to infinite buffers without restraint can easily be paralysed by excessive input. With correctly implemented backpressure, on the other hand, every demand is strictly enforced: The publisher only delivers as many elements as were actually requested. This prevents malicious or faulty sources from overloading the system by producing data en masse. Instead, resource consumption remains predictable and controlled, increasing the stability and security of the entire pipeline.

Where does that fit in the JDK?

• Namespace : java.util.concurrent.Flow – this package bundles the core building blocks of the reactive API. It provides interfaces for publishers, subscribers, subscriptions, and processors, and thus forms the standard framework for reactive streams in the JDK. This ensures that all implementations adhere to the same contracts and can be seamlessly integrated.
• Roles : The roles can be distinguished as follows: A publisher creates and delivers data to its subscribers, a subscriber consumes this data and processes it further, the subscription forms the contract between the two and regulates the backpressure signals, among other things, and the processor takes on a dual role by receiving data as a subscriber and at the same time passing on transformed or filtered data to the next consumer as a publisher.
• Thetypes in the Flow API are generic and require careful and thread-safe handling. Generic means that Publisher, Subscriber, and Processor are always instantiated with specific type variables, so that the data type remains consistent throughout the pipeline. At the same time, strict attention must be paid to thread safety during implementation, as signals such as onNext or onError can arrive from different threads and must be synchronised correctly. As an introduction, the JDK provides a reference implementation with the SubmissionPublisher, which is sufficient for simple scenarios and already takes into account essential concepts such as backpressure. In production systems, however, developers often fall back on their own or adapted processor classes to precisely implement specific requirements for buffering, transformation, error handling or performance.

1.2 Push vs. Pull: Reactive Streams vs. Java Streams

Aspect	Java Streams (Pull)	Reactive Streams / Flow (Push)
**Data direction	Consumer pulls data	Producer pushes data
Execution	type. synchronously in the calling thread	asynchronous, often via executor/threads
**Backpressure	non-existent	central concept (request(s))
Lifetime	finite, terminal operation	Potentially infinite, subscription-based
**Error	Exceptions in the caller	onError‑signal in the stream
Demolition	End of Source	cancel() of the subscription

Minimal examples

Pull (Java Streams) – the consumer sets the pace:

Note: SubmissionPublisher is a convenient way to get started, but not a panacea (limited tuning options). Custom processor‑implementations give you full control over backpressure, buffer, and concurrency.

xmlCopy

@Test  void test001() {    var data = List.of(1, 2, 3, 4, 5);    int sum = data.stream()        .filter(n -> n % 2 == 1)        .mapToInt(n -> n * n)        .sum();    System.out.println("sum = " + sum);  }//Push (flow) – the producer delivers, the subscriber signals demand:import java.util.concurrent.Flow.*;import java.util.concurrent.SubmissionPublisher;  @Test  void test002() {    class SumSubscriber implements Flow.Subscriber<Integer> {      private Flow.Subscriptions;      private int sum = 0;      @Override public void onSubscribe(Flow.Subscription s) { this.s = s; s.request(1); }      @Override public void onNext(Integer item) { sum += item; s.request(1); }      @Override public void onError(Throwable t) { t.printStackTrace(); }      @Override public void onComplete() { System.out.println("sum = " + sum); }    }    try (var pub = new SubmissionPublisher<Integer>()) {      pub.subscribe(new SumSubscriber());      for (int i = 1; i<= 5; i++) pub.submit(i); asynchronous push    }  }

1.3 Typical Use Cases

• Telemetry & Logging: In modern systems, events are continuously created, such as metrics or log entries. These streams can be so high frequency that they cannot be permanently stored or transmitted in their raw form. This is where a processor comes into play, which first caches the events and then combines them in batches, for example, 100 events each. In this way, the flood of data can be effectively throttled and persisted in manageable portions, preventing overloading of individual components.
• Message ingestion: In many architectures, message and event systems, such as Kafka, classic message queues, or HTTP endpoints, form the input layer for data. These can be modelled as publishers who continuously provide new events. A processor then takes over the validation and possible enrichment, such as checking mandatory fields, adding additional metadata or transforming them into a uniform format. Only then does the enriched and checked data pass on to the subscriber, who takes over the actual persistence, for example, by writing it to a database or data warehouse. In this way, a clear separation of responsibility is achieved, allowing errors to be detected and addressed early in the data stream.
• IoT/streams from sensors: In practice, sensors often deliver data streams at highly fluctuating speeds. Some sensors transmit data in millisecond intervals, while others transmit data only every few seconds or minutes. If these flows are passed on unchecked to a central consumer, there is a risk of buffer overflow and thus instability in the overall system. With backpressure, however, the consumer can actively control how many measured values they take. In scenarios where the rate exceeds the processing capacity, additional strategies can be employed: either older values are discarded (DropStrategy) or only the most recent value is passed on (LatestStrategy). This keeps the system stable, allowing the consumer to always work with up-to-date and relevant data without being overwhelmed by data avalanches.
• UIEvents & RateControl: In user interfaces, event streams with very high frequencies quickly arise, for example, when a user types a search query and an event is triggered for every keystroke. Without regulation, these events would be passed on to theBackendService unhindered, which could overload both the network connections and the servers. In this case, a special DebounceProcessor ensures that only the last event is forwarded within a short period of time. This avoids unnecessary inquiries, and the user still receives up-to-date suggestions promptly. This technique effectively prevents the UI from being flooded with data and, at the same time improves the perceived performance and responsiveness of the application.
• Security: For safety-critical applications, it is imperative that each stage of a data pipeline is clearly delineated and only communicates via defined contracts. This prevents the creation of an unbridled shared state that could lead to inconsistencies or security gaps. Another aspect is the possibility of interrupting data flows cleanly at any time. A cancel() method allows a subscriber to terminate the subscription if they determine that the source is faulty, provides too much data, or presents a potential security threat. In this way, resources can be freed up immediately, and the pipeline remains stable and resilient even under adverse conditions.

2. Overview: java.util.concurrent.Flow

After explaining the motivation and the basic differentiation from Java Streams in the first chapter, we will now deal with the actual framework that the JDK provides for reactive streams. With the java.util.concurrent.The Flow package, a small but powerful API, has been provided since Java 9, describing the essential roles and contracts of Reactive Streams. These interfaces are deliberately kept minimal, allowing them to be used directly for simple use cases as well as forming the basis for more complex libraries.

The focus is on four roles that define the entire lifecycle of publishers and subscribers:Publisher ,Subscriber ,Subscription andProcessor.

Apublisher is the source of the data. He produces elements and makes them accessible to his subscribers. He may only send data if it has been explicitly requested beforehand. This prevents him from sending events uncontrollably, which in the worst case, fizzle out into nowhere or cause memory problems.

Asubscriber is the counterpart to this: they receive the data that the publisher provides and process it further. To avoid being overloaded, the subscriber informs the publisher of the number of items they can include at a given time as part of the subscription.

Thesubscription is the link between the publisher and the subscriber. It is created at the moment of registration (the so-called subscribe process) and handed over to the subscriber. From then on, it controls the flow of data by providing methods such as request(long n) and cancel(). This controls demand and, simultaneously, enables a clean interruption of data transmission.

Finally, aprocessor combines both roles: it is a subscriber itself because it receives data from a publisher, but at the same time, it is also a publisher because it passes on the transformed or filtered data. This dual function is the great strength of the processor: it enables the construction of complex pipelines from simple building blocks, where transformation, filtering, or aggregation are clearly separated and encapsulated.

2.1 Life cycle of the signals

The flow within the Flow API follows a clear order. As soon as a subscriber logs in to a publisher, the publisher first calls the subscriber’s onSubscribe method and passes the subscription. Only when the subscriber requests elements via this subscription – such as request(1) or request(10) – does the publisher begin to deliver data. Each delivered item is delivered by a call to onNext.

If all data has been sent or the stream is terminated for other reasons, the publisher signals the end with onComplete. If, on the other hand, an error occurs, it is reported via onError . This precise signal semantics ensures that every data flow is either completed regularly or terminated with an error message. A subscriber, therefore, always knows exactly what state the data source is in and can react accordingly.

2.2 Backpressure basic principle

A central concept of Flow is the so-called backpressure. It prevents publishers from flooding their subscribers with data. Instead, the subscribers themselves control how many items they want to retrieve. For example, a subscriber can initially request only a single element with request(1) and only ask for the next one after it has been processed. Similarly, it is possible to order larger quantities, such as request(50), to increase throughput. This model gives consumers complete control and ensures a stable balance between production and consumption.

Backpressure is not only a technical detail, but a decisive criterion for the robustness of reactive systems. Without this control, publishers could deliver events unchecked, overloading resources such as CPU or memory. Backpressure, on the other hand, can also be used to implement complex scenarios such as batching, prioritisation or flow control across thread and system boundaries.

The Flow API in the JDK defines the elementary building blocks of a reactive pipeline. Publisher, subscriber, subscription and processor together form a clear model that supports both small experiments and upscaled systems. The signal lifecycle and backpressure mechanisms ensure that the data streams are controlled, traceable and stable. This lays the foundation for delving deeper into the role of the processor in the following chapters and shedding light on its concrete applications.

3. Focus on Flow.Processor <I,O>

After the four roles were briefly described in the previous chapter, the focus now shifts to the processor. It occupies a key position in the flow API because it fulfils two roles simultaneously: it isa subscriber to its upstream, i.e., it receives data from a publisher, and it is alsoa publisher to its downstream, i.e., it passes on transformed or filtered data to other subscribers. This makes it act like a hinge in a chain of processing steps.

A processor is always provided with two type parameters: <I, O>. The first type describes which elements it receives from the upstream, and the second type describes which elements it passes on to the downstream. This allows any transformations to be mapped – for example, from raw sensor data to validated measured values or from text messages to structured objects.

3.1 Contract & Responsibilities

Implementing a processor means that you must abide by the rules of both subscribers and publishers. This includes, in particular, the following aspects:

• Pay attention to signal flow: The processor must handle all signals it receives from the upstream device wholly and correctly. These include the onSubscribe, onNext, onError, and onComplete methods. Each of these signals fulfils a particular function in the lifecycle of a data stream: onSubscribe initiates the subscription and hands over control of the data flow, onNext signals the actual data, onError signals the occurrence of an error, and onComplete marks the regular end of the stream. It is essential that the processor strictly adheres to these semantics: An error may only be reported once and unambiguously, and no further data may follow after the onComplete event occurs. Only through these clear rules does processing remain consistent, predictable and reliable for all components involved.
• Respect backpressure: As a subscriber, the processor is obligated to adhere strictly to the rules of the backpressure model. This means that it may only receive and process exactly as many elements as it has previously actively requested from the upstream via its subscription – for example, by making a call such as request(1) or request(10). This prevents him from being overrun by a publisher who is too fast. At the same time, as a publisher, he has the responsibility to pass on this logic unchanged to his downstream. He is therefore not allowed to forward more elements to his subscribers than they have explicitly requested. The processor thus acts as a pass-through for the demand signals, ensuring that the entire chain, from publisher to processor to subscriber, remains in balance and that there are no overloads or data losses.
• Manage resources cleanly: A processor must also ensure that the resources it manages can be released cleanly at all times. In particular, this includes the fact that subscriptions can be actively terminated, for example, by calling cancel() if a subscriber loses interest or processing is to be interrupted for other reasons. It is equally vital that there are no memory or thread leaks: Open queues must be emptied or closed, background threads must be properly terminated, and executor services must be shut down again. Only through this consistent management can the system remain stable, performant and free of creeping resource damage in the long term.
• Do not swallow errors: If an error occurs, it must be consistently and transparently passed on to the downstream system via the onError method. This ensures that downstream subscribers are also clearly informed about the status of the data stream and can react – for example, through logging, retry mechanisms or the targeted termination of their own processing. If, on the other hand, an error is tacitly ignored, there is a risk of inconsistencies that are difficult to understand. Equally problematic is the uncontrolled throwing of unchecked exceptions, as they are not caught cleanly in the context of reactive pipelines; thus, entire processing chains can enter an undefined state. Clean error propagation is therefore a central quality feature of every processor implementation.

Type Variables & Invariants

The type parameters can be used to determine precisely what kind of data a processor can process. For example, a processor<string, integer> could receive lines of text and extract numbers from them, which it then passes on. These types must be strictly adhered to a subscriber who expects integers must not receive strings unexpectedly. The genericity in the Flow API ensures that errors of this kind are already noticeable during compilation.

Additionally, the invariant holds that a processor must behave like both a correct subscriberand a correct publisher. He is therefore not only responsible for the transformation, but also for the proper transfer of tax information, such as demand and cancellation.

Chain position: between upstream & downstream

In practice, a processor is rarely used in isolation, but as an intermediate link in a chain. An upstream publisher delivers data that the processor receives, transforms or filters and then forwards to downstream subscribers. This creates flexible pipelines that can be expanded or exchanged depending on requirements.

For example, a logging processor can be placed between a data source and the actual consumer, also to record all elements. A processor can also be used to buffer data before it is passed on in larger batches. The position in the middle allows different aspects, such as transformation, control and observation, to be separated from each other and made modular.

The Flow.Processor is the link in reactive pipelines. Through his dual function as subscriber and publisher, he takes responsibility for the correct processing of signals, the implementation of transformations and compliance with the backpressure rules. Its type parameters <I,O> also ensure that data flows remain strictly typed and thus reliable. In the following chapters, we will show how to implement your own processor classes and which patterns have proven themselves in practice.

4. Practical introduction with the JDK

Now that the concepts and contracts around the Flow.Processor have been presented in detail, it is time to start with a practical implementation in the JDK. Fortunately, Java has provided a reference implementation for a publisher since version 9, with the SubmissionPublisher, which is ideal for initial experiments. It relieves you of many details, such as thread management and internal buffering, allowing you to concentrate fully on the data flow.

The SubmissionPublisher is designed to distribute data asynchronously to its subscribers. In the background, it works with an executor that packages and distributes the individual signals, such as onNext, onError and onComplete, into separate tasks. By default, it uses the ForkJoinPool.commonPool(), but it can also pass its own executor. For small experiments, the standard configuration is usually sufficient; however, in production scenarios, it is worthwhile to adapt the executor and buffer sizes to your specific requirements.

4.1 SubmissionPublisher in a nutshell

The SubmissionPublisher class implements the Publisher interface and is therefore an immediately usable source of data. It provides the submit(T item) method, which is used to add new items to the stream. These items are then automatically distributed to all registered subscribers. In this way, the publisher assumes the role of an asynchronous dispatcher, passing incoming data to any number of subscribers.

An essential detail is that the submit method does not block, but buffers the elements internally and then distributes them. However, if the buffer limits are reached, submit can block or even throw an IllegalStateException if the system is under too much load. That’s why it’s crucial to find the right balance between publisher speed, buffer size, and subscriber demand.

xmlCopy

import java.util.concurrent.Flow.*;import java.util.concurrent.SubmissionPublisher;  @Test  void test003() {    class PrintSubscriber        implements Flow.Subscriber<Integer> {      private Flow.Subscription subscription;      @Override      public void onSubscribe(Flow.Subscription subscription) {        System.out.println("Register: 1 element/req" );        this.subscription = subscription;        subscription.request(1); Request first element      }      @Override      public void onNext(Integer item) {        System.out.println("Receive: " + item);        subscription.request(1); request another after each item      }      @Override      public void onError(Throwable t) {        t.printStackTrace();      }      @Override      public void onComplete() {        System.out.println("Done!");      }    }    try (SubmissionPublisher<Integer> publisher = new SubmissionPublisher<>()) {      publisher.subscribe(new PrintSubscriber());      for (int i = 1; i<= 5; i++) {        System.out.println(" send element = " + i );        publisher.submit(i);      }    }  }

4.2 Simple pipeline without its own processor

To understand how SubmissionPublisher works, it’s worth taking a simple example. This creates a publisher that sends numbers from 1 to 5 to a subscriber. The subscriber receives the data and outputs it to the console. This example shows the basic flow of onSubscribe, onNext, and onComplete.

In this program, the publisher asynchronously generates five values, which are transmitted to the subscriber one after the other. The subscriber explicitly requests an additional element each time, creating a controlled and stable flow of data. Finally, the publisher signals with onComplete that no more data will follow.

4.3 Limitations of SubmissionPublisher

Even though the SubmissionPublisher is very helpful for getting started, it quickly reaches its limits in more complex scenarios. For example, it offers only limited possibilities for configuring the backpressure behaviour. While the buffer size is customizable, it doesn’t directly support more complex strategies, such as dropping or latest-value. Additionally, the standard executor is not suitable for all applications, particularly when high latency requirements or strict thread isolation are necessary.

Another point is that SubmissionPublisher is primarily intended for learning purposes and simple applications. In productive systems, people usually rely on their own publisher and processor implementations or on established reactive frameworks such as Project Reactor or RxJava, which are based on the Flow API and provide additional operators.

The SubmissionPublisher is a handy place to start to see the concepts of the Flow API in action. It shows how publishers and subscribers interact, how backpressure works and how signals are processed. At the same time, however, it also becomes clear that for more complex or productive scenarios, in-house processor implementations or external libraries are indispensable. In this way, he forms the bridge between the theoretical foundations and the first practical steps in working with reactive pipelines in the JDK.

5. Sample Processor with Code Examples

Now that the foundations for your own implementation of a processor have been laid, it makes sense to look at typical patterns. They show how concrete use cases can be implemented and form the basis for more complex pipelines. Here are some simple but commonly used processor types, each with sample code.

5.1 MapProcessor <I,O> Transformation

The MapProcessor is the simplest and at the same time one of the most useful processors. It takes on the role of a transformation: incoming data is changed with a function and then passed on. For example, strings can be converted to their length or numbers can be squared.

xmlCopy

@Test  void mapProcessor_usage_minimal()      throws Exception {    final CountDownLatch done = new CountDownLatch(1);    final Function<String,Integer> mapper = s -> Integer.parseInt(s) * 2;//1) Publisher (source)    try (SubmissionPublisher<String> publisher = new SubmissionPublisher<>();//2) MapProcessor (String -> Integer)         MapProcessor<String,Integer> map = new MapProcessor<>(mapper)) {//3) Target Subscribers (Spend Only)      map.subscribe(          new Flow.Subscriber<>() {            private Flow.Subscriptions;            @Override            public void onSubscribe(Flow.Subscription subscription) { this.s = subscription; s.request(1); }            @Override            public void onNext(Integer item) { out.println("receive = " + item); s.request(1); }            @Override            public void onError(Throwable t) { t.printStackTrace(); done.countDown(); }            @Override            public void onComplete() { done.countDown(); }          });//Concatenation: Publisher -> MapProcessor      publisher.subscribe(map);//Send data      publisher.submit("1");      publisher.submit("2");      publisher.submit("3");      publisher.close();//signal end Wait briefly for completion (asynchronous)      done.await();    }  }  /**   * Minimal processor that applies a Function<I,O>.   * No own threads/executor – uses the defaults of the SubmissionPublisher.   */  static class MapProcessor<I,O>      extends SubmissionPublisher<O>      implements Flow.Processor<I,O> {    private final Function<I,O> mapper;    private Flow.Subscription subscription;    MapProcessor(Function<I,O> mapper) {      super();      this.mapper = mapper;    }    @Override    public void onSubscribe(Flow.Subscription subscription) { this.subscription = subscription; subscription.request(1); }    @Override    public void onNext(I item) { submit(mapper.apply(item)); subscription.request(1); }    @Override    public void onError(Throwable throwable) { closeExceptionally(throwable); }    @Override    public void onComplete() { close(); }  } 

This allows data streams to be elegantly transformed without adjusting the rest of the pipeline.

5.2 FilterProcessor – Filtering Data

Another typical pattern is filtering. Only those elements that meet a certain condition are passed on. This is especially helpful when large amounts of data are generated, of which only a fraction is relevant.

xmlCopy

@Test  void filterEvenNumbers()      throws Exception {    Predicate<Integer> even = n -> {      out.println("Filter: " + n);      return n % 2 == 0;    };    try (SubmissionPublisher<Integer> publisher = new SubmissionPublisher<>()) {      FilterProcessor<Integer> filter = new FilterProcessor<>(even);      CountDownLatch done = new CountDownLatch(1);      CopyOnWriteArrayList<Integer> received = new CopyOnWriteArrayList<>();      Flow.Subscriber<Integer> subscriber = new Flow.Subscriber<>() {        private Flow.Subscription sub;        @Override        public void onSubscribe(Flow.Subscription subscription) {          this.sub = subscription;          sub.request(1);        }        @Override        public void onNext(Integer item) {          received.add(item);          sub.request(1);        }        @Override        public void onError(Throwable throwable) { done.countDown(); }        @Override        public void onComplete() { done.countDown(); }      };      publisher.subscribe(filter);      filter.subscribe(subscriber);      List.of(1, 2, 3, 4, 5, 6).forEach(publisher::submit);      publisher.close();      //Short waiting time for asynchronous processing      if (!done.await(1, TimeUnit.SECONDS)) {        throw new TimeoutException("Processing did not finish in time");      }      out.println("received = " + received);      assertEquals(List.of(2, 4, 6), received);    }  }  static class FilterProcessor<T>      extends SubmissionPublisher<T>      implements Flow.Processor<T,T> {    private final Predicate<T> predicate;    private Flow.Subscription subscription;    public FilterProcessor(Predicate<T> predicate) { this.predicate = predicate; }    @Override    public void onSubscribe(Flow.Subscription subscription) {      this.subscription = subscription;      subscription.request(1);    }    @Override    public void onNext(T item) {      if (predicate.test(item)) { submit(item); }      subscription.request(1);    }    @Override    public void onError(Throwable throwable) { closeExceptionally(throwable); }    @Override    public void onComplete() { close(); }  }

With this processor, data streams can be reduced in a targeted manner and made more relevant.

5.3 BatchingProcessor – Collect and Share

Sometimes it doesn’t make sense to pass on every single element right away. Instead, they want to collect data and pass it on in blocks. The BatchingProcessor does exactly this job.

This processor is useful for processing data efficiently in blocks, for example when writing to a database.

xmlCopy

import java.util.*;@Test  void testBatch()      throws InterruptedException {    List<List<Integer>> received = new ArrayList<>();    CountDownLatch done = new CountDownLatch(1);    try (SubmissionPublisher<Integer> publisher = new SubmissionPublisher<>();         BatchingProcessor<Integer> batching = new BatchingProcessor<>(3)) {      publisher.subscribe(batching);      batching.subscribe(new Flow.Subscriber<>() {        private Flow.Subscription subscription;        @Override        public void onSubscribe(Flow.Subscription s) {          this.subscription = s;          s.request(1);        }        @Override        public void onNext(List<Integer> batch) {          System.out.println("onNext - batch = " + batch);          received.add(batch);          subscription.request(1);        }        @Override        public void onError(Throwable t) {          done.countDown();          throw new AssertionError("Unexpected error", t);        }        @Override        public void onComplete() {          System.out.println("onComplete...");          done.countDown();        }      });      for (int i = 1; i<= 10; i++) {        System.out.println("submitting i = " + i);        publisher.submit(i);      }      publisher.close();      done.await();    }    assertThat(received)        .containsExactly(            List.of(1, 2, 3),            List.of(4, 5, 6),            List.of(7, 8, 9),            List.of(10)        );  }  class BatchingProcessor<T>      extends SubmissionPublisher<List<T>>      implements Flow.Processor<T,List<T>> {    private final List<T> buffer = new ArrayList<>();    private final int batchSize;    private Flow.Subscription subscription;    public BatchingProcessor(int batchSize) { this.batchSize = batchSize; }    @Override    public void onSubscribe(Flow.Subscription subscription) {      this.subscription = subscription;      subscription.request(1);    }    @Override    public void onNext(T item) {      buffer.add(item);      if (buffer.size() >= batchSize) {        submit(new ArrayList<>(buffer));        buffer.clear();      }      subscription.request(1);    }    @Override    public void onError(Throwable throwable) { closeExceptionally(throwable); }    @Override    public void onComplete() {      if (!buffer.isEmpty()) {        submit(new ArrayList<>(buffer));        buffer.clear();      }      close();    }  }

5.4 ThrottleProcessor Throttling

Especially with high-frequency sources, it is necessary to limit the rate of data passed on. A ThrottleProcessor can be implemented in such a way that it only forwards one element at certain time intervals and discards the rest.

xmlCopy

@Test  void throttle_allows_items_only_in_interval()      throws Exception {    try (SubmissionPublisher<Integer> publisher = new SubmissionPublisher<>();         ThrottleProcessor<Integer> throttle = new ThrottleProcessor<>(50)) { // 50 ms interval      CollectingSubscriber<Integer> subscriber = new CollectingSubscriber<>(3); we expect 3 elements      //Build a pipeline: publisher -> throttle -> subscriber      publisher.subscribe(throttle);      throttle.subscribe(subscriber);      //Send Items: 1 (should through), 2/3 (throttled), 4 (through), 5 (throttled), 6 (through)      publisher.submit(1);      Thread.sleep(10);      publisher.submit(2);      Thread.sleep(10);      publisher.submit(3);      Thread.sleep(60); //> 50ms: Next may go through      publisher.submit(4);      Thread.sleep(10);      publisher.submit(5);      Thread.sleep(60);      publisher.submit(6);      publisher.close(); //Completion of the source      assertTrue(subscriber.await(2, TimeUnit.SECONDS),                 "Timeout while waiting for throttled items");      var received = subscriber.getReceived();      logger().info("Received {} items", received);      assertEquals(List.of(1, 4, 6), received);    }  }  //---- Processor ----  static class ThrottleProcessor<T>      extends SubmissionPublisher<T>      implements Flow.Processor<T,T>, HasLogger {    private final long intervalMillis;    private Flow.Subscription subscription;    private long lastEmission = 0;    public ThrottleProcessor(long intervalMillis) {      this.intervalMillis = intervalMillis;    }    @Override    public void onSubscribe(Flow.Subscription subscription) {      this.subscription = subscription;      subscription.request(1);    }    @Override    public void onNext(T item) {      logger().info("onNext {}", item);      long now = System.currentTimeMillis();      if (now - lastEmission >= intervalMillis) {        logger().info("submit item " + item);        submit(item);        lastEmission = now;      } else {        logger().info("< intervalMillis-skippingitem{}",item);     }     subscription.request(1);   }   @Override   publicvoidonError(Throwablethrowable){closeExceptionally(throwable);}   @Override   publicvoidonComplete(){close();} } //----simplecollector-subscriber---- staticclassCollectingSubscriber<T>      implements Flow.Subscriber<T> , HasLogger {    private final List<T> received = new ArrayList<>();    private final CountDownLatch latch;    private Flow.Subscription subscription;    CollectingSubscriber(int expectedCount) {      this.latch = new CountDownLatch(expectedCount);    }    @Override    public void onSubscribe(Flow.Subscription subscription) {      this.subscription = subscription;      subscription.request(Long.MAX_VALUE); unbounded demand    }    @Override    public void onNext(T item) {      logger().info("onNext {}", item);      received.add(item);      latch.countDown();    }    @Override    public void onError(Throwable throwable) { /* no-op for test */ }    @Override    public void onComplete() { /* no-op for test */ }    List<T> getReceived() { return received; }    boolean await(long timeout, TimeUnit unit)        throws InterruptedException {      return latch.await(timeout, unit);    }  } 

This pattern can be used, for example, to prevent a UI event stream from generating too many updates uncontrollably.

The sample processors presented here are fundamental building blocks for designing reactive pipelines in the JDK. With transformation, filtering, batching and throttling, many typical requirements are covered. The following chapters will focus on implementing backpressure in practice and discuss which strategies have proven successful in operation.

6. Backpressure in practice

After presenting concrete sample processors in the previous chapter, we will now focus on one of the central topics in the Flow API: Backpressure. While the theory sounds simple – the subscriber determines the number of elements they can process – in practice, it turns out that the right implementation is crucial for stability, efficiency and robustness.

6.1 Strategies for dealing with backpressure

Backpressure prevents a publisher from flooding its subscribers with data. Nevertheless, there are different strategies for coping with demand:

• Bounded buffering: In this strategy, the publisher only keeps a clearly defined, limited number of elements in an internal buffer. Once this buffer is filled, there are several possible reactions: the submit() method can block until there is space again, or it throws an exception to indicate the overload. In this way, an uncontrolled growth of the memory is prevented. This model is particularly suitable in scenarios where a controlled and predictable data rate is required – for example, in systems that are only allowed to process a certain number of requests per second or in hardware interfaces with limited processing capacity.
• Dropping: When overloaded, new elements are deliberately discarded instead of being cached in an infinite buffer. This strategy makes sense, especially where not every single event is critical and the loss of individual data points remains tolerable. Typical examples are telemetry data, which is generated at a high frequency anyway, or UI events such as mouse movements or keystrokes, where only the current state is relevant for further processing. Dropping keeps the system responsive and resource-efficient even under extreme load, as it doesn’t try to funnel every single event through the pipeline at all costs.
• Latest-Value: In the Latest-Value strategy, old elements are not collected in a buffer, but are consistently discarded as soon as a new value arrives. Only the most recently delivered element is stored in each case, so that the subscriber only receives the latest version the next time it is retrieved or asked. This procedure is beneficial when values change continuously. Still, only the last state is relevant for processing – for example, in cases where sensor data, such as temperature or position coordinates, is involved. This relieves the subscriber because he does not have to work through all the intermediate results, but can continue working directly with the latest information.
• Blocking: In the blocking strategy, the publisher actively waits for demand to be signalled again by the subscriber before delivering further elements. In practice, this means that the calling thread blocks until a corresponding request is made. This approach is comparatively easy to implement and makes the process calculable at first glance, as it ensures that no more data is produced than was requested. However, this approach has a significant drawback: blocked threads are a scarce resource in highly concurrent systems, and when many publishers block at the same time, entire thread pools can become clogged. Therefore, blocking should only be used in very special scenarios, such as when the data rate is low anyway or when controlling the exact data flow is more important than maximum parallelism.

6.2 Demand Policy: Batch vs. Single Retrieval

Subscribers can determine for themselves how many items they call up at a time. There are two common approaches:

• Single retrieval ( request(1) ): After each item received, exactly one more is requested. This approach is simple and provides maximum control, but it generates a large number of signals and can become inefficient at very high rates.
• Batch retrieval ( request(n) ): The subscriber requests several elements at once, such as request(32). This reduces the signal load, allowing the publisher to deliver items in blocks more efficiently. However, the subscriber must then make sure that he can handle these batches.

In practice, both approaches are often combined, depending on whether low latency or high throughput is the primary concern.

6.3 Monitoring, Tuning and Capacity Planning

A decisive success factor in the use of backpressure is monitoring. Without measuring points, it remains unclear whether a system is running stably or is already working at its limits. The following key figures are helpful:

• Queue depth: The queue depth refers to the number of elements currently in the internal buffer that the subscriber has not yet retrieved. It is a direct indicator of whether the publisher is rushing away from the consumer or whether demand is in balance with production.
• Latency: Latency refers to the time elapsed between the creation of an element and its final processing by the subscriber. It is a key measure of the system’s responsiveness: high latency indicates that elements are jamming in buffers or processing steps are taking too long. In contrast, low latency means a smooth data flow.
• Throughput (items per second): Throughput indicates the number of items that can be processed per unit of time. It’s a measure of the entire pipeline’s performance: high throughput means that publishers, processors, and subscribers can work together efficiently and meet demand. If the throughput drops below the expected level, this is an indication of a bottleneck – whether due to too small buffers, too slow processing in the subscriber, or insufficient parallelisation. Throughput is therefore an important key figure for realistically estimating the capacity of a system and making targeted optimisations.

This data can be used to identify bottlenecks and take appropriate measures, such as adjusting the batch size, increasing the buffers, or introducing additional processor stages.

Concurrency & Execution Context

A central feature of the Flow API is its close integration with concurrent processing. Unlike classic Java streams, which usually run synchronously in the same thread, reactive streams almost always involve an asynchronous interaction of several threads. Therefore, it is crucial to understand the role of the execution context in detail.

7.1 Executor Strategies

The Flow API itself does not specify on which threads the signals are processed. Instead, the implementations determine which executor they use. The SubmissionPublisher uses the ForkJoinPool.commonPool() by default, but also allows you to include your own executor. This is extremely important in practice, as the choice of the executor can change the behaviour of the entire system:

• ACachedThreadPool can enable very high parallelism through its dynamic generation and reuse of threads. It grows indefinitely when there are many tasks at the same time, and reduces again when there is less work. This makes it exceptionally flexible and well-suited for unpredictable load peaks. However, this dynamic can also lead to uncontrollable behaviour: If thousands of threads are suddenly created, the context switching load increases significantly, which has an adverse effect on latency. It is therefore less suitable for latency-critical scenarios, as response times can become unpredictable due to the administrative overhead and the potentially high number of threads.
• AFixedThreadPool works with a fixed number of threads, which are defined when the pool is created. In this way, it creates predictable limits for concurrency and prevents uncontrolled growth of the thread count. This makes it particularly suitable for scenarios in which resources such as CPU cores or memory are to be used in a clearly limited and predictable manner. However, there is a disadvantage when overloaded: If there are more tasks than threads available, queues form. These lead to increasing latencies and can become problematic in critical systems if the queue grows unchecked. It is therefore essential to consciously dimension the pool and possibly use mechanisms such as backpressure or rejection policies to absorb overload situations in a controlled manner.
• TheForkJoinPool is optimised for fine-grained, recursive, and highly parallelizable tasks. He relies on a work-stealing procedure in which worker threads actively search for functions that are in the queues of other threads when they themselves have no more work. This achieves a high utilisation of the available threads and makes efficient use of computing time. This model excels above all in CPU-intensive calculations, which can be broken down into many small, independent tasks – for example, in divide-and-conquer algorithms or parallelised data analyses. The ForkJoinPool, on the other hand, is less suitable for blocking operations such as network access, file system or database queries. When a worker thread blocks, it becomes unavailable for work stealing, making the pool scarce, and the model’s efficiency suffers. In such cases, it is advisable to outsource blocking work to separate executors with a suitable thread policy or to rely on the virtual threads introduced by Loom, which are more tolerant of blocking operations. Additionally, in specific scenarios, the use of ForkJoinPool is applicable. ManagedBlocker can help to mitigate the adverse effects of blocking operations by allowing the pool size to be temporarily extended.

Choosing the right executor is, therefore, a key architectural point that directly impacts throughput, latency, and stability.

7.2 Reentrancy traps and serialization of signals

Another critical issue is the question of how the onNext, onError and onComplete signals are delivered. The Reactive Streams specification specifies that these signals must be called sequentially and not simultaneously, so they must be serialised. Nevertheless, in practice, parallel calls arise due to your own implementations or clumsy synchronisation. This quickly leads to errors that are difficult to reproduce, the so-called “Heisenbugs”.

Therefore, a processor or subscriber must ensure that he is not confronted with onNext calls from several threads at the same time. This can be achieved through synchronisation, using queues or by delegation to an executor, which processes the signals one after the other.

7.3 Virtual Threads (Loom) vs. Reactive

With Project Loom, Java has introduced a new model: Virtual Threads. These enable the creation of millions of lightweight threads that efficiently support blocking operations. This blurs the classic line between blocking and non-blocking code. The question, therefore, arises: Do we still need reactive streams with complex backpressure at all?

The answer is: Yes, but differentiated. While virtual threads make it easier to handle multiple simultaneous operations, they do not automatically resolve the issue of uncontrolled data production. Even with Virtual Threads, a subscriber must be able to specify the number of elements they can process. Backpressure, therefore, remains an important concept. However, virtual threads can help simplify the implementation of subscriber logic, as blocking processing steps now scale much better.

Error & Completion Scenarios

In every reactive data stream, the question arises as to how to deal with errors and the regular closing. The Flow API provides clear semantics for this: Each stream ends either with a successful completion (onComplete) or with an error (onError). This simple but strict model ensures that a subscriber knows exactly what state the data flow is in at all times.

8.1 Semantics of onError and onComplete

The onError and onComplete methods mark the endpoint of a data stream. Once one of these methods has been called, no further data may be sent through onNext. This contract is crucial because it guarantees consistency: a subscriber can rest assured that they won’t have to process any new items at the end.

• onComplete means that all elements have been successfully submitted and the source has been exhausted. Examples include reading a file or playing back a finite list of messages.
• onError signals that an error occurred during processing. This error is passed on to the subscriber as an exception so that he can react in a targeted manner – for example, through logging, restarts, or emergency measures.

It is important to note that onError and onComplete are mutually exclusive. So it must never happen that a stream first breaks off with an error and then signals a regular end.

8.2 Restarts and Retry Mechanisms

In many use cases, it is not sufficient to simply terminate a stream at the first error. Instead, they want to try to resume processing. Classic examples are network requests or database access, which can fail temporarily.

There is no built-in retry functionality in the flow API itself, but it can be implemented at the processor level. For example, after an error, a processor could decide to restart the request or skip the faulty record. It is important to handle the state clearly: A new onSubscribe may only take place when a new data stream is started. For more complex scenarios, patterns such as circuit breaker or retry with exponential backoff are ideal.

A simple example could be a subscriber who raises a counter on the first error and retries the faulty operation a maximum of three times. Only if an error still occurs after these retries does it finally pass the error on to the downstream:

xmlCopy

class RetrySubscriber<T> implements Flow.Subscriber<T> {    private final Flow.Subscriber<? super T> downstream;    private int retries = 0;    private static final int MAX_RETRIES = 3;    private Flow.Subscription subscription;    RetrySubscriber(Flow.Subscriber<? super T> downstream) {        this.downstream = downstream;    }    @Override    public void onSubscribe(Flow.Subscription subscription) {        this.subscription = subscription;        downstream.onSubscribe(subscription);        subscription.request(1);    }    @Override    public void onNext(T item) {        downstream.onNext(item);        subscription.request(1);    }    @Override    public void onError(Throwable t) {        if (retries< MAX_RETRIES){           retries++;           System.out.println("Retry"+retries+"aftererror:"+t.getMessage());           subscription.request(1);Tryingtokeepgoing       }else{           downstream.onError(t);       }   }   @Override   publicvoidonComplete(){       downstream.onComplete();   }}

This simple pattern is of course greatly simplified, but it clarifies the principle of a repeated attempt in the event of temporary errors.

8.3 Idem potency and exactly-once processing

A central problem with restarts is determining whether operations can be carried out more than once. Many systems are only robust if their operations areidempotent – that is, they produce the same result when executed multiple times. An example is writing a record with a specific key to a database: even if this process is repeated, the final state remains the same.

If idempotency is absent, there is a risk of double processing, which can lead to incorrect results or inconsistencies. Therefore, it is good practice to prefer idempotent operations or to ensure that data is not processed multiple times through additional mechanisms, such as transaction IDs, deduplication, or exactly-once processing.

A simple example: Suppose a processor processes orders and passes each order with a unique ID to an OrderService, which stores it. Even if the same record passes through the processor twice, the ID ensures that it only exists once in the service. This means that the operation remains idempotent.

A small Java example with aprocessor and JUnit5Test might look like this:

xmlCopy

import org.junit.jupiter.api.Test;import static org.junit.jupiter.api.Assertions.*;import java.util.*;import java.util.concurrent.*;import java.util.function.Function;  @Test  void processor_enforces_idempotence_by_filtering_duplicates()      throws Exception {    OrderService svc = new OrderService();    try (SubmissionPublisher<Order> pub = new SubmissionPublisher<>()) {      DeduplicateProcessor<Order,String> dedup = new DeduplicateProcessor<>(Order::id);      pub.subscribe(dedup);      dedup.subscribe(new SinkSubscriber(svc));      pub.submit(new Order("order-1", "Order A"));      pub.submit(new Order("order-1", "Order A (Duplicate)"));      pub.submit(new Order("order-2", "Order B"));      pub.close();      Thread.sleep(100); asynchronous drain    }    assertEquals(2, svc.size());    assertNotNull(svc.get("order-1"));    assertNotNull(svc.get("order-2"));  }  //Domain  record Order(String id, String payload) { }  //Processor: filters duplicates based on a key (idempotency in the pipeline)  static class DeduplicateProcessor<T,K>      extends SubmissionPublisher<T>      implements Flow.Processor<T,T> , HasLogger {    private final Function<T,K> keyExtractor;    private final Set<K> seen = ConcurrentHashMap.newKeySet();    private Flow.Subscription subscription;    DeduplicateProcessor(Function<T,K> keyExtractor) { this.keyExtractor = keyExtractor; }    @Override    public void onSubscribe(Flow.Subscription s) {      this.subscription = s;      s.request(1);    }    @Override    public void onNext(T item) {      logger().info("Receive {}", item);      if (seen.add(keyExtractor.apply(item))) {        logger().info("no duplicate - submitting");        submit(item);      } else {        logger().info("Duplicate - ignore");      }      subscription.request(1);    }    @Override    public void onError(Throwable t) { closeExceptionally(t); }    @Override    public void onComplete() { close(); }  }  //Subscriber writes to a simple "DB"  static class OrderService {    private final Map<String,Order> store = new ConcurrentHashMap<>();    void save(Order o) { store.put(o.id(), o); }    Order get(String id) { return store.get(id); }    int size() { return store.size(); }  }  static class SinkSubscriber      implements Flow.Subscriber<Order>, HasLogger {    private final OrderService svc;    private Flow.Subscriptions;    SinkSubscriber(OrderService svc) { this.svc = svc; }    @Override    public void onSubscribe(Flow.Subscription s) {      this.s = s;      s.request(1);    }    @Override    public void onNext(Order item) {      logger().info("Save {}", item);      svc.save(item);      s.request(1);    }    @Override    public void onError(Throwable t) { }    @Override    public void onComplete() { }  }

The test shows that despite the publication of order-1 being repeated twice, only one instance is sent downstream, thanks toDeduplicateProcessor. The idempotency logic is thus in theprocessor , not in the subscriber.

Interoperability & Integration

A key feature of the Flow API is its openness to integration. The API itself is minimal and only defines the basic contracts for publisher, subscriber, subscription and processor. This allows it to be used both standalone and combined with other reactive libraries.

9.1 Integration with Reactive Frameworks

Many established reactive frameworks, such asProject Reactor ,RxJava, orAkka Streams, are based on the same fundamental ideas as the Flow API and are closely tied to the official Reactive Streams specification. They offer either direct adapters or interfaces that can be used to integrate publishers, subscribers and, in particular, your own processor implementations. This means that self-developed building blocks can be easily embedded in complex ecosystems without being tied to a specific framework. A central example is the helper class FlowAdapters in the JDK, which provides conversion methods between java.util.concurrent.Flow and org.reactivestreams.Publisher. This enables seamless integration of your own JDK-based publishers or processors with Reactor or RxJava operators without breaking the contracts. In practice, this means that a processor implemented with Flow can be integrated directly into a Reactor flux or fed from an RxJava observable, allowing existing pipelines to be expanded or migrated step by step.

A simple example shows the use of FlowAdapters. This example can also be secured with a small JUnit5 test:

xmlCopy

import org.junit.jupiter.api.Test;import static org.junit.jupiter.api.Assertions.*;import java.util.concurrent.*;import java.util.concurrent.Flow.*;import java.util.concurrent.FlowAdapters;public class FlowAdapterTest {    @Test    void testFlowAdapterIntegration() throws Exception {        try (SubmissionPublisher<String> jdkPublisher = new SubmissionPublisher<>()) {            // Flow -> Reactive Streams            org.reactivestreams.Publisher<String> rsPublisher = FlowAdapters.toPublisher(jdkPublisher);            //back to Flow            Publisher<String> flowPublisher = FlowAdapters.toFlowPublisher(rsPublisher);            List<String> results = Collections.synchronizedList(new ArrayList<>());            flowPublisher.subscribe(new Subscriber<>() {                private subscriptions;                @Override public void onSubscribe(Subscription s) { this.s = s; s.request(1); }                @Override public void onNext(String item) { results.add(item); s.request(1); }                @Override public void onError(Throwable t) { fail(t); }                @Override public void onComplete() { }            });            jdkPublisher.submit("Hello World");            jdkPublisher.close();            Thread.sleep(100); Wait asynchronously            assertEquals(List.of("Hello World"), results);        }    }}

This test shows that a SubmissionPublisher can be successfully turned into a Reactive Streams publisher and back again via FlowAdapters. The message “Hello world” reaches the subscriber correctly at the end.

9.2 Use in classic applications

The Flow API can also be used sensibly in non-reactive applications. For example, data streams from a message queue or a websocket can be distributed to internal services via a SubmissionPublisher. Legacy systems that have previously relied on polling benefit from distributing and processing events in real-time. This enables modern, reactive architectures to be integrated into existing applications incrementally.

A simple example illustrates the connection with a legacy service:

xmlCopy

import java.util.concurrent.SubmissionPublisher;import java.util.concurrent.Flow;//Simulates an existing legacy service that was previously addressed via pollingclass LegacyService {    public void process(String msg) {        System.out.println("LegacyService handles: " + msg);    }}//Subscriber who passes incoming events to the legacy serviceclass LegacyServiceSubscriber implements Flow.Subscriber<String> {    private Flow.Subscriptions;    private final LegacyService service;    LegacyServiceSubscriber(LegacyService service) { this.service = service; }    @Override public void onSubscribe(Flow.Subscription s) { this.s = s; s.request(1); }    @Override public void onNext(String item) { service.process(item); s.request(1); }    @Override public void onError(Throwable t) { t.printStackTrace(); }    @Override public void onComplete() { System.out.println("Stream completed"); }}public class LegacyIntegrationDemo {    public static void main(String[] args) {        LegacyService legacy = new LegacyService();        try (SubmissionPublisher<String> pub = new SubmissionPublisher<>()) {            pub.subscribe(new LegacyServiceSubscriber(legacy));            //Example: instead of polling, a MessageQueue delivers new messages            pub.submit("Message 1 from MQ");            pub.submit("Message 2 from MQ");        }    }}

This small program simulates use in a classic application: Instead of polling, messages from an external source are immediately forwarded to the LegacyService , which can continue to work unchanged. The Flow API serves as a bridge between modern event processing and existing logic.

9.3 Bridge to Java Streams

A common question is how the Flow API compares to or even connects to classic Java Streams (the Stream API). While streams typically process finite amounts of data in the pull model, flow works with potentially infinite sequences in the push model. Nevertheless, a combination is possible: A processor could cache incoming data and feed it into a Java stream. Conversely, the results of a stream can be brought back into a reactive context via a publisher. This creates a bridge between batch-oriented and event-driven processing.

A small JUnit5 example demonstrates this bridge: Here, elements are transferred to a stream via a SubmissionPublisher and then summed.

xmlCopy

import org.junit.jupiter.api.Test;import static org.junit.jupiter.api.Assertions.*;import java.util.concurrent.*;import java.util.*;public class FlowToStreamTest {    @Test    void testFlowToStreamIntegration() throws Exception {        try (SubmissionPublisher<Integer> publisher = new SubmissionPublisher<>()) {            List<Integer> buffer = Collections.synchronizedList(new ArrayList<>());            //Subscriber collects data in a list            publisher.subscribe(new Flow.Subscriber<>() {                private Flow.Subscriptions;                @Override public void onSubscribe(Flow.Subscription s) { this.s = s; s.request(1); }                @Override public void onNext(Integer item) { buffer.add(item); s.request(1); }                @Override public void onError(Throwable t) { t.printStackTrace(); }                @Override public void onComplete() { }            });            Publish events            publisher.submit(1);            publisher.submit(2);            publisher.submit(3);            publisher.close();            //short pause to wait for asynchronous collection            Thread.sleep(100);            //Transition to Java Stream: Calculate Sum            int sum = buffer.stream().mapToInt(i -> i).sum();            assertEquals(6, sum);        }    }}

In this example, the list acts as a buffer and allows the transition from the asynchronous push model (flow) to the synchronous pull model (stream). In this way, both worlds can be elegantly combined. However, the topic is far more extensive and will be examined in detail in a separate article.

Result

The Flow API injava.util.Concurrent shows how Java provides a clean, standardised basis for reactive data streams. While Java Streams are intended for finite, synchronous pull scenarios, Flow addresses the challenges of continuous, infinite and asynchronous data sources. Core concepts includebackpressure , precise signal semantics (onNext, onError, onComplete), and the dual function of theFlow.Processor create the basis for stable, extensible pipelines.

The patterns shown – from map and filter processors to batching and throttling – make it clear that robust processing chains can be built with little code. Supplemented by retry strategies, idempotency and interoperability with established reactive frameworks, a toolbox is created that is suitable for simple learning examples as well as for upscaled systems.

This makes it clear: If you work with event streams in Java, you can’t get around the Flow API. It bridges the gap between classic streams, modern reactive programming and future developments such as virtual threads. It is crucial to consistently adhere to the principles of backpressure, clean use of resources and clear signal guidance. This results in pipelines that are not only functional, but also durable, high-performance and safe.

]]>Design PatternJavahttps://svenruppert.com/posts/signal-via-sse-data-via-rest-a-vaadin-demonstration-in-core-java/Wed, 03 Sep 2025 13:05:56 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/signal-via-sse-data-via-rest-a-vaadin-demonstration-in-core-java/1. Introduction 1.1 Motivation: Event-driven updating without polling In classic web applications, the pull principle still dominates: Clients repeatedly make requests to the server to detect changes. This polling is simple, but it leads to unnecessary load on the server and network side, especially if the data stock changes only sporadically. Server-Sent Events (SSE) is a standardised procedure that allows the server to signal changes to connected clients actively. This avoids unnecessary requests, while updates reach the interface promptly.<![CDATA[

1. Introduction

1.1 Motivation: Event-driven updating without polling

In classic web applications, the pull principle still dominates: Clients repeatedly make requests to the server to detect changes. This polling is simple, but it leads to unnecessary load on the server and network side, especially if the data stock changes only sporadically.Server-Sent Events (SSE) is a standardised procedure that allows the server to signal changes to connected clients actively. This avoids unnecessary requests, while updates reach the interface promptly.

1. 1. Introduction
   1. 1.1 Motivation: Event-driven updating without polling
   2. 1.2 Objectives and delimitation
   3. 1.3 Overview of the demonstration scenario
2. 2. Conceptual Foundations
   1. 2.1 Server-Sent Events (SSE)
   2. 2.2 REST as a transport channel for payload data
   3. 2.3 Vaadin Flow: Server-Side UI Model and Push
3. 3. Architecture of the demonstrator
   1. 3.1 Component Overview
   2. 3.2 Communication relationships
   3. 3.3 Runtime Environment and Assumptions
4. 4. Data and event model
   1. 4.1 Record Structure
   2. 4.2 Event Types and Semantics
   3. 4.3 Sequence numbering and idempotent reloading
5. 5. REST/SSE Server with In-Process CLI
   1. 5.1 Operating concept of the CLI
   2. 5.2 Append Log and Consistency Considerations
   3. 5.3 SSE Broadcast: Trigger on Successful Input
   4. 5.4 Optional optimisations
6. 6. Vaadin Flow Integration (without JavaScript)
   1. 6.1 Server-Side SSE Client
   2. 6.2 UI Synchronisation and Push
   3. 6.3 Interaction patterns
   4. 6.4 Delta Retrieval and Full Recall
7. 7. Implementation – REST SSE Server
   1. 7.1 REST Server
   2. 7.2 Entry
   3. 7.3 SseHandler
   4. 7.4 DataHandler
8. 8. Implementation – Vaadin Flow UI
   1. 8.1 DashboardView
   2. 8.2 DataClient
   3. 8.3 Entry
   4. 8.4 SseClientService
   5. 8.5 UiBroadcaster
9. 9. Summary
   1. 9.1 Evaluation of the “Signal-per-SSE, Data-per-REST” pattern
   2. 9.2 Didactic benefits and reusability

1.2 Objectives and delimitation

This article aims to demonstrate the basic functionality of SSE in interaction with a Vaadin Flow application. The focus is on the separation of signalling and data retrieval: While SSE is used exclusively for notifications, the actual retrieval of the new data is done via REST endpoints. Security aspects such as authentication or authorisation, as well as persistent data storage, are deliberately excluded to emphasise the core idea clearly.

The code is under the following URL on github:

https://github.com/Java-Publications/Blog---Vaadin---How-to-consume-SSE-from-a-REST-Service

1.3 Overview of the demonstration scenario

The demonstrator consists of three components:

• REST/SSE server with CLI input : New data is entered directly from the console during runtime and stored in simple in-memory storage. Each input immediately triggers an SSE signal.
• SSE signal : The server sends an event after each input that tells the clients that new data is available.
• Vaadin Flow application : The UI registers as an SSE client, displays notifications about new data and allows users to reload and display them in a targeted manner via REST retrieval.

This constellation provides a clear and comprehensible example of integrating SSE into a server-side Java UI framework. The separation between signal and data allows for a robust and easy-to-understand architecture that is suitable for a wide range of real-time scenarios.

2. Conceptual Foundations

2.1 Server-Sent Events (SSE)

Server-sent events (SSE) are a standardised procedure for continuously transmitting messages from the server to the client. Communication is unidirectional: the server sends, the client receives. Technically, SSE is based on a persistent HTTP connection with the MIME typetext/event-stream. Messages consist of simple lines of text and are interpreted by the browser or a client framework.

The advantages of SSE include the simplicity of implementation, automatic client-side reconnect, and good integration into existing HTTP infrastructures. Limitations result from the restriction to UTF-8 text and the lack of possibility of bidirectional communication. However, for pure notifications and status messages, SSE is usually more efficient and robust than more complex alternatives such as WebSockets or Long Polling.

2.2 REST as a transport channel for payload data

REST-based endpoints have established themselves as the standard for transmitting structured or binary data. They are stateless, easily scalable and benefit from established infrastructure such as caching and monitoring. In this demonstration scenario, REST is used to provide the actual data that the client explicitly requests after a notification via SSE. This separation ensures clear responsibilities: SSE signals that something has changed; REST offers the content.

2.3 Vaadin Flow: Server-Side UI Model and Push

Vaadin Flow is a server-side Java framework for web application development. The UI logic runs on the server, while the browser only takes care of the display. Changes in server state are transmitted to the client via a synchronised communication channel. For the integration of SSE, it is particularly relevant that Vaadin works withpush : Server-side events can be transferred directly to the interface. This allows the reception of SSE signals to be elegantly combined with UI updates without the need for client-side JavaScript.

3. Architecture of the demonstrator

3.1 Component Overview

The demonstrator consists of three main components. The first component is theREST/SSE server. It is responsible for providing an endpoint for entering new data, storing this data in memory and simultaneously sending signals to the connected clients via SSE as soon as the data stock changes. In addition, it provides REST endpoints through which the clients can retrieve the current data.

The second component is thein-process CLI , which is embedded directly in the server process. It enables manual entry of new data rows via the console during runtime. Each input is immediately stored as a data record in the in-memory. At the same time, this process triggers an event that is forwarded to the connected clients via SSE.

The third component is theVaadin Flow application. It acts as a client that receives the signals sent via SSE and informs users that new data is available. If desired, the application can load this data specifically via the provided REST endpoints and display it directly in the user interface. This covers the entire process from input to signalling to visibility in the UI.

3.2 Communication relationships

The architecture follows the patternsignal via SSE, data via REST. As soon as the CLI receives a new input, the server stores the entry in memory and sends a corresponding SSE update to all connected clients. The Vaadin application receives this signal, informs the user of the availability of new data, and provides an option for targeted retrieval. Via a subsequent REST request, the Vaadin application requests the data and updates the user interface.

3.3 Runtime Environment and Assumptions

Some deliberately simplified framework conditions apply to the demonstration. The server and Vaadin application run locally on the same host, but in separate processes and typically on different ports, such as 8080 for the server and 8081 for the UI. This simulates a realistic separation of the systems without the need for a complex infrastructure.

In addition, the CORS policy is set in such a way that access is possible without restriction. This open configuration is only for the sake of simplifying the demonstration, as security considerations are not taken into account in this scenario.

The data is stored exclusively in the working memory. Each record entered remains available only during runtime and is lost after the process ends. Persistence mechanisms such as databases or file systems are deliberately not used in this example to focus entirely on the interaction of CLI input, SSE signal and REST retrieval.

This architecture is deliberately kept simple to demonstrate the functionality of SSE in combination with REST and Vaadin Flow in a clear and comprehensible way.

4. Data and event model

4.1 Record Structure

In the demonstration scenario, new data is generated by entering lines of text in the CLI. Each entry is stored in the server in the form of a simple data record. This consists of a consecutive sequence number that is incremented with each input, a timestamp that documents the time of entry, and the actual content, i.e. the text entered by the operator. It is implicitly assumed that both the REST/SSE server and the Vaadin application are based on the same system time. This is the only way to compare the timestamps directly without the need for additional synchronisation mechanisms. This structure is deliberately kept minimalist to ensure traceability and to focus on the interaction of the components.

4.2 Event Types and Semantics

Communication between server and client takes place via events that are transmitted in SSE format. The focus is on theupdate event, which is always sent when a new record is added. At a minimum, the current sequence number is transmitted as the data load of this event, so that the client can detect whether there are any new entries for it. Optionally, aninit event could also be used, which signals the start state when a new connection is established. Other event types, such as reset or snapshot, can be provided for advanced scenarios, but are not required for demonstration.

4.3 Sequence numbering and idempotent reloading

The sequence number plays a central role in the consistency between server and client. It enables the retrieval of only the data records added since the last known number via REST. This allows clients to remain correctly synchronised even if they have missed one or more SSE signals. This makes the reload idempotent: a new call with the samesince specification always returns the same result set, regardless of how often it is repeated. This principle facilitates fault tolerance and ensures a stable processing chain.

5. REST/SSE Server with In-Process CLI

5.1 Operating concept of the CLI

The server process has an integrated command line interface that can be used to enter new records during runtime. Each input corresponds to a single line of text that is taken directly into the in-memory. This simple operating concept allows the data source to be controlled manually and thus trigger targeted events, which are then signalled to the clients via SSE.

5.2 Append Log and Consistency Considerations

The entries are stored in the form of an append log: Each new record is added to the end of the existing list. The sequential sequence number ensures that the sequence can be clearly determined. This simple structure ensures that all clients can consistently reload the same state when needed. More complex mechanisms, such as transactions or locks, are not required for demonstration.

5.3 SSE Broadcast: Trigger on Successful Input

As soon as a new record is added to memory, the server immediately generates an SSE event of typeupdate. This event is sent out to all connected clients and contains at least the current sequence number. This enables all registered recipients to recognise that new data is available. The actual content is not transmitted, but is reserved for later retrieval via REST.

5.4 Optional optimisations

For a demonstration scenario, it is sufficient to forward each new event immediately. However, in more realistic environments, optimisations can be helpful. This includes, for example, combining several fast inputs into a combined update signal to reduce the number of messages transmitted. Similarly, regular ping comments can be used to stabilise the connection and ensure that inactive clients are detected and removed. These measures increase robustness, but are not necessary for functional demonstration. We will explore these concepts further in another article, specifically when we discuss the open-source URL shortener project.

6. Vaadin Flow Integration (without JavaScript)

6.1 Server-Side SSE Client

The Vaadin Flow application operates a server-side SSE client that is permanently connected to the SSE endpoint of the REST server. This allows the application to receive signals independently of client-side JavaScript and process them directly on the server side. If a connection is interrupted, a reconnect attempt is automatically started so that the UI remains continuously informed about the current status.

6.2 UI Synchronisation and Push

Since Vaadin Flow works on the principle of a server-side UI model, external events must be integrated into the UI thread. This is done via synchronised accesses, which ensure that changes are executed consistently and thread-safe. In combination with Vaadin Push, incoming SSE signals can be converted directly into visible updates. Users notice the update immediately, without the need for a manual refresh. We deliberately disregard the special case of a complete page reload at this point.

6.3 Interaction patterns

The Vaadin interface is designed to only display a hint at first, rather than automatically loading all new data upon arrival of an incoming SSE signal. This can be done in the form of a notification or by activating a button. Only through a conscious action by the user is the new data retrieved via REST and integrated into the interface. In addition, the user can selectively decide which detailed data is of interest. This interaction pattern illustrates the separation of signalling and data retrieval and makes the functionality particularly clear for demonstration purposes.

6.4 Delta Retrieval and Full Recall

When reloading the data via REST, a distinction can be made between two modes: A complete retrieval always loads the entire data set. In contrast, a delta retrieval retrieves only those entries that have been added since the last known sequence number. If no sequence number is available, the entire database is not transferred, but only the last n messages are loaded. This procedure is sufficient for the demonstration, as it facilitates traceability. In scenarios closer to production, delta retrieval still offers advantages in terms of efficiency and bandwidth.

7. Implementation – REST SSE Server

7.1 REST Server

TheRestServer bundles the entire server functionality based oncom.sun.net.httpserver.HttpServer. It initialises the three endpoints of the demonstrator (/sse,/data,/health) and manages the server resources required for SSE. These include the number of currently connected output streams (sseClients), an in-memory append log of the data records, and the execution services required at runtime (a Scheduled Executor for Keep pings and a Connection Executor for long-lived SSE connections).

During operation, the server starts a CLIThread that receives rows fromstdin and inserts them into the log as new records. Each entry is given a monotonic sequence number and a time stamp; an SSEUpdate is then sent to all connected clients. TheRestServer also provides helper functions for CORSHeader, standardised answers, query parsing, and the output of the SSE form. The since(long) andlastN(int) methods map the REST reading paths: They either return all entries after a specific sequence number or the lastn entries for initial synchronisation if no sequence is yet known.

xmlCopy

package com.svenruppert.rest;public class RestServer {  public static final int DEFAULT_LAST_N = 20;  public static final String PATH_SSE = "/sse";  public static final String PATH_DATA = "/data";  public static final String PATH_HEALTH = "/health";  public static final String CONTENT_TYPE = "text/plain; charset=utf-8";  public static final int PORT = 8080;  public static final long PING_INTERVAL_MILLIS = 30_000L;  protected static final String ACCESS_CONTROL_ALLOW_ORIGIN = "Access-Control-Allow-Origin";  private final HttpServer http;  private final Set<OutputStream> sseClients = ConcurrentHashMap.newKeySet();  private final CopyOnWriteArrayList<Entry> store = new CopyOnWriteArrayList<>();  private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2);  private final ExecutorService connectionExecutor = Executors.newCachedThreadPool();  private final AtomicLong seq = new AtomicLong(0);  public RestServer(int port)      throws IOException {    http = HttpServer.create(new InetSocketAddress(port), 0);    http.createContext(PATH_SSE, new SseHandler(this));    http.createContext(PATH_DATA, new DataHandler(this));    http.createContext(PATH_HEALTH, ex -> respond(ex, 200, "OK"));    http.setExecutor(connectionExecutor);  }  ---Main---  public static void main(String[] args)      throws Exception {    RestServer srv = new RestServer(PORT);    Runtime.getRuntime().addShutdownHook(new Thread(srv::stop));    srv.start();  }  public AtomicLong getSeq() {    return seq;  }  public Set<OutputStream> getSseClients() {    return sseClients;  }  public ScheduledExecutorService getScheduler() {    return scheduler;  }  public ExecutorService getConnectionExecutor() {    return connectionExecutor;  }  public void writeEvent(OutputStream os, String event, String data)      throws IOException {    os.write(sseFormat(event, data));    os.flush();  }  public void writeComment(OutputStream os, String comment)      throws IOException {    os.write(("# " + comment + "\n\n").getBytes(StandardCharsets.UTF_8));    os.flush();  }  public byte[] sseFormat(String event, String data) {    String msg = "event: " + event + "\n" + "data: " + data + "\n\n";    return msg.getBytes(StandardCharsets.UTF_8);  }  --- Utils ---  public void addCors(HttpExchange ex) { ex.getResponseHeaders().add(ACCESS_CONTROL_ALLOW_ORIGIN, "*"); }  public void respond(HttpExchange ex, int code, String body)      throws IOException { respond(ex, code, body, CONTENT_TYPE); }  public void respond(HttpExchange ex, int code, String body, String contentType)      throws IOException {    Headers h = ex.getResponseHeaders();    h.add("Content-Type", contentType);    byte[] bytes = body.getBytes(StandardCharsets.UTF_8);    ex.sendResponseHeaders(code, bytes.length);    try (OutputStream os = ex.getResponseBody()) {      os.write(bytes);    }  }  public Map<String,List<String>> parseQuery(URI uri) {    Map<String,List<String>> map = new LinkedHashMap<>();    String query = uri.getRawQuery();    if (query == null || query.isEmpty()) return map;    for (String pair : query.split("&")) {      int idx = pair.indexOf('=');      String key = idx > 0 ? decode(pair.substring(0, idx)) : decode(pair);      String val = idx > 0&& pair.length() > idx + 1 ? decode(pair.substring(idx + 1)) : "";      map.computeIfAbsent(key, k -> new ArrayList<>()).add(val);    }    return map;  }  public String decode(Strings) { return URLDecoder.decode(s, StandardCharsets.UTF_8); }  public String first(List<String> list) { return (list == null || list.isEmpty()) ? null : list.get(0); }  --- Start / Stop ---  public void start() {    http.start();    System.out.println("REST/SSE server running on http://localhost:" + PORT);    CLI thread for in-process input    Thread cli = new Thread(this::cliLoop, "cli-loop");    cli.setDaemon(true);    cli.start();  }  public void stop() {    try {      http.stop(0);    } catch (Exception ignored) {    }    try {      scheduler.shutdownNow();    } catch (Exception ignored) {    }    try {      connectionExecutor.shutdownNow();    } catch (Exception ignored) {    }    for (OutputStream os : sseClients) {      try {        os.close();      } catch (IOException ignored) {      }    }    sseClients.clear();  }  ---CLI---  protected void cliLoop() {    System.out.println("CLI ready. Type in lines of text. 'exit' terminates the server.");    try (BufferedReader br = new BufferedReader(new InputStreamReader(System.in, StandardCharsets.UTF_8))) {      String line;      while ((line = br.readLine()) != null) {        if (line.equalsIgnoreCase("exit")) {          System.out.println("End server...");          stop();          break;        }        if (line.isBlank()) {          System.out.println("(blank line ignored)");          continue;        }        appendNew(line);      }    } catch (IOException e) {      System.err.println("CLI exited: " + e.getMessage());    }  }  private void appendNew(String text) {    long n = seq.incrementAndGet();    Entry e = new Entry(n, Instant.now(), text);    store.add(e);    System.out.println("[APPEND] " + e);    broadcastUpdate(s);  }  private void broadcastUpdate(long highestSeq) {    String payload = Long.toString(highestSeq);    byte[] bytes = sseFormat("update", payload);    List<OutputStream> dead = new ArrayList<>();    for (OutputStream os : sseClients) {      try {        os.write(bytes);        os.flush();      } catch (IOException e) {        dead.add(os);      }    }    if (!dead.isEmpty()) sseClients.removeAll(dead);  }  public List<Entry> since(long s) {    List<Entry> out = new ArrayList<>();    for (Entry e : store) if (e.seq() > s) out.add(e);    return out;  }  public List<Entry> lastN(int n) {    int size = store.size();    if (n<= 0) return List.of();    int from = Math.max(0, size - n);    return new ArrayList<>(store.subList(from, size));  }}

7.2 Entry

Entrymodels a single record as a record with the fieldsseq (sequence number),ts (timestamp) andtext (payload from the CLI). The record is immutable and therefore well-suited for concurrent read accesses. ThetoString()representation is deliberately kept simple and outputs the three fields in one line; it is used for text-based transmission via the /data endpoint and supports simple parsing on the client side.

javaCopy

packagecom.svenruppert.rest;---Model---publicrecordEntry(longseq,Instantts,Stringtext){ @NotNull @Override publicStringtoString(){   returnseq+"|"+DateTimeFormatter.ISO_INSTANT.format(ts)+"|"+text; }}

7.3 SseHandler

TheSseHandler implements the SSE endpoint/sse. When called, it sets up the HTTP response for atext/event stream , inserts the client’s output stream into the managed set of SSE connections, and sends an initial event to confirm the successful connection. A periodic Keep‑Alive mechanism (Ping Comments) keeps the connection open and supports the detection of aborted clients. The handler works closely with the helper functions provided by theRestServer (writing individual events or comments, accessing schedulers, and connection management).

If there is a write error or the remote station is closed, the handler removes the connection from the set of active streams and cleans up associated resources. In combination with the server’s broadcasting, this creates a resilient but straightforward push model for signals via newly available data.

javaCopy

packagecom.svenruppert.rest.handler;---SIZE---publicrecordSseHandler(RestServerrestServer)   implementsHttpHandler{ @Override publicvoidhandle(HttpExchangeex)     throwsIOException{   if(!" GET".equals(ex.getRequestMethod())){     restServer.respond(ex,405,"Method Not Allowed");     return;   }   restServer.addCors(ex);   Headersh=ex.getResponseHeaders();   h.add("Content-Type","text/event-stream; charset=utf-8");   h.add("Cache-Control","no-cache");   h.add("Connection","keep-alive");   ex.sendResponseHeaders(200,0);   finalOutputStreamos=ex.getResponseBody();   restServer.getSseClients().add(os);   InitialEvent   restServer.writeEvent(os,"init","ready");   // Keep-Alive Pings   ScheduledFuture<?>pinger=restServer.getScheduler().scheduleAtFixedRate(()->{     try{       restServer.writeComment(os,"ping");     }catch(IOExceptione){/* will be closed below */}   },RestServer.PING_INTERVAL_MILLIS,RestServer.PING_INTERVAL_MILLIS,TimeUnit.MILLISECONDS);   //Keep blocking open until client/OS closes   restServer.getConnectionExecutor().execute(()->{     try(os){       //NOP: we only write at events, otherwise Ping will keep the line open       //Wait for an exception to occur/OS to close       //(an explicit read/write loop is not necessary here)       Thread.currentThread().join();     }catch(InterruptedExceptionignored){     }catch(Exceptionignored){     }finally{       pinger.cancel(true);       restServer.getSseClients().remove(os);       try{         os.close();       }catch(IOExceptionignored){       }     }   }); }}

7.4 DataHandler

TheDataHandler provides the REST endpoint/data and implements the two intended read variants. Depending o sequence number is known,on the Query parameters, it returns all entries with a sequence number greater than a given the lastn entries by default (configurable viaDEFAULT_LAST_N or thelastN parameter). The response is generated astext/plain, with each record output on its own line.

The handler also validates the QueryParameters and produces consistent error messages for invalid inputs. Together with theSseHandler, it maps the separation of signaling (SSE) and payload retrieval (REST) and allows a deterministic, idempotent reload process on the client side.

xmlCopy

package com.svenruppert.rest.handler;---DATA---public record DataHandler(RestServer restServer)    implements HttpHandler {  @Override  public void handle(HttpExchange ex)      throws IOException {    if (!" GET".equals(ex.getRequestMethod())) {      restServer.respond(ex, 405, "Method Not Allowed");      return;    }    restServer.addCors(ex);    Map<String,List<String>> q = restServer.parseQuery(ex.getRequestURI());    String sinceStr = restServer.first(q.get("since"));    String lastNStr = restServer.first(q.get("lastN"));    List<Entry> result;    if (sinceStr != null&& !sinceStr.isBlank()) {      long s;      try {        s = Long.parseLong(sinceStr);      } catch(NumberFormatException nfe) {        restServer.respond(ex, 400, "since must be a number");        return;      }      result = restServer.since(s);    } else {      int n = RestServer.DEFAULT_LAST_N;      if (lastNStr != null&& !lastNStr.isBlank()) {        try {          n = Integer.parseInt(lastNStr);        } catch(NumberFormatException nfe) {          restServer.respond(ex, 400, "lastN must be a number");          return;        }      }      result = restServer.lastN(n);    }    Output as text/plain, one line per entry: seq|ISO-TS|text    StringBuilder sb = new StringBuilder();    for (Entry e : result) {      sb.append(e.toString()).append('\n');    }    restServer.respond(ex, 200, sb.toString(), RestServer.CONTENT_TYPE);  }}

8. Implementation – Vaadin Flow UI

8.1 DashboardView

TheDashboardViewis the central view of the demonstration and is integrated via the routedashboard. It presents the current data in agrid and provides the option to reload with a button. When attached, the view registers with theUiBroadcaster and binds a listener to theSseClientService. If an “update” signal arrives, the view informs the users (status bar, notification) and activates the reload button. The actual retrieval only takes place after user action: Either new entries are loaded since the last known sequence (lastSeq) or, if no sequence is yet available, the lastn messages. After successful retrieval,lastSeqis updated, and the grid is updated consistently. With the detach, the view moves away from the broadcaster and deregisters the listener.

xmlCopy

package com.svenruppert.flow.views.dashboard;@Route(value = DashboardView.ROUTE, layout = MainLayout.class)public class DashboardView    extends Composite<VerticalLayout>    implements HasLogger {  public static final String ROUTE = "dashboard";  ---Configuration---  private static final String BASE = "http://localhost:8090"; Server Base  private static final String SSE_URL = BASE + "/sse";  private static final String DATA_URL = BASE + "/data";  private static final int DEFAULT_LAST_N = 20;  ---UI---  private final Grid<Entry> grid = new Grid<>(Entry.class, false);  private final Button fetchBtn = new Button("fetch data");  private final Span status = new Span("Waiting for events ...");  --- client services ---  private final DataClient dataClient = new DataClient(DATA_URL);  private final SseClientService sseClient = new SseClientService(SSE_URL);  private final AtomicBoolean hasNew = new AtomicBoolean(false);  ---Condition---  private volatile Long lastSeq = zero; last confirmed sequence  public DashboardView() {    getContent().setSizeFull();    grid.addColumn(Entry::seq).setHeader("Seq").setAutoWidth(true).setFlexGrow(0);    grid.addColumn(e -> e.ts().toString()).setHeader("Timestamp").setAutoWidth(true).setFlexGrow(0);    grid.addColumn(Entry::text).setHeader("Text").setFlexGrow(1);    fetchBtn.setEnabled(false);    fetchBtn.addClickListener(e -> fetch());    getContent().add(status, fetchBtn, grid);    addAttachListener(ev -> {      UI ui = ev.getUI();      UiBroadcaster.register(ui);      SseClientService.Listener l = (type, data) -> {        if ("update".equals(type)) {          try {            long seq = Long.parseLong(data.trim());            mark only; Fetch decides on Delta/lastN            hasNew.set(true);            UiBroadcaster.broadcast(() -> {              fetchBtn.setEnabled(true);              status.setText("New data available (seq=" + seq + ") – please retrieve.");              Notification.show("New data available", 1200, Notification.Position.TOP_CENTER);            });          } catch (NumberFormatException ignore) {            Fallback: Hint without seq            hasNew.set(true);            UiBroadcaster.broadcast(() -> {              fetchBtn.setEnabled(true);              status.setText("New data available – please retrieve.");            });          }        }      };      sseClient.addListener(l);      addDetachListener(ev2 -> {        sseClient.removeListener(l);        UiBroadcaster.unregister(ui);      });    });  }  private void fetch() {    fetchBtn.setEnabled(false);    List<Entry> toShow;    if (lastSeq != null) {      toShow = dataClient.fetchSince(lastSeq);    } else {      toShow = dataClient.fetchLastN(DEFAULT_LAST_N);    }    if (!toShow.isEmpty()) {      lastSeq = toShow.getLast().seq(); Latest Bookmark    }    UI ui = UI.getCurrent();    if (ui != null) {      effectively final      ui.access(() -> {        if (!toShow.isEmpty()) {          List<Entry> current = new ArrayList<>(grid.getListDataView().getItems().toList());          current.addAll(toShow);          grid.setItems(current);          status.setText("Data loaded(" + current.size() + " entries).");        } else {          status.setText("No new entries.");        }        hasNew.set(false);      });    }  }}

8.2 DataClient

TheDataClientencapsulates the REST‑communication with the /data endpoint. It offers two read paths:fetchSince(long since) for delta fetches and fetchLastN(int n) for initial synchronisation. The answers are expected astext/plainand converted to a list ofEntry(format:seq|ISOTS|text, one line per record). By encapsulating the HTTPDetails, the view remains slim; Changes to the transport format or timeouts can be adjusted centrally without touching the UICode.

xmlCopy

package com.svenruppert.flow.views.dashboard;--- REST client for /data ---public final class DataClient {  private final HttpClient http = HttpClient.newBuilder()      .connectTimeout(Duration.ofSeconds(5)).build();  private final String baseUrl;  public DataClient(String baseUrl) {    this.baseUrl = baseUrl;  }  //Server format: seq|ISO-TS|text per line  private static List<Entry> parse(String body)      throws IOException {    List<Entry> out = new ArrayList<>();    if (body == null || body.isBlank()) return out;    String[] lines = body.split("\\R");    for (String line : lines) {      if (line.isBlank()) continue;      String[] parts = line.split("\\|", 3);      if (parts.length< 3)continue;     longseq =Long.parseLong(parts[0]);     Instantts =Instant.parse(parts[1]);     Stringtext =parts[2];     out.add(newEntry(seq,ts,text));   }   returnout; } publicList<Entry> fetchSince(long since) {    String url = baseUrl + "?since=" + since;    return fetch(url);  }  public List<Entry> fetchLastN(int n) {    String url = baseUrl + "?lastN=" + n;    return fetch(url);  }  private List<Entry> fetch(String url) {    try {      HttpRequest req = HttpRequest.newBuilder(URI.create(url))          .timeout(Duration.ofSeconds(5)). GET().build();      HttpResponse<String> resp = http.send(req, HttpResponse.BodyHandlers.ofString());      if (resp.statusCode() != 200) return List.of();      return parse(resp.body());    } catch (Exception e) {      return List.of();    }  }}

8.3 Entry

Entryrepresents a single record with a monotonic sequence number, timestamp, and message content. The immutable record is well suited for display in the grid and for concurrency in the view. In the Vaadin demonstration,Entry serves as a lightweight transport and display object that is generated directly from the RESTresponse and passed on unchanged.

javaCopy

packagecom.svenruppert.flow.views.dashboard;publicrecordEntry(longseq,Instantts,Stringtext){}

8.4 SseClientService

TheSseClientServicerepresents the server-side SSEClient of the Vaadin‑application. It maintains a long-lived HTTP‑connection to/sse, parses incoming events in the formattext/event-stream and distributes them to registered listeners in the application. After disconnections, a reconnect is automatically attempted. The service‑boundary is deliberately simple: Events are passed on astype/data(especiallyevent: update,data: <seq>). The View then decides whether and how to reload. This preserves the separation between signaling (SSE) and data transmission (REST).

xmlCopy

package com.svenruppert.flow.views.dashboard;//--- SSE client (server-side) ---public final class SseClientService {  private final HttpClient http = HttpClient.newBuilder()      .connectTimeout(Duration.ofSeconds(5)).build();  private final String url;  private final List<Listener> listeners = new CopyOnWriteArrayList<>();  private volatile boolean running = false;  public SseClientService(String url) {    this.url = Objects.requireNonNull(url);  }  private static void sleep(long ms) {    try {      Thread.sleep(ms);    } catch (InterruptedException ignored) {    }  }  public void addListener(Listener l) {    listeners.add(l);    ensureLoop();  }  public void removeListener(Listener l) {    listeners.remove(l);  }  private synchronized void ensureLoop() {    if (running) return;    running = true;    Thread t = new Thread(this::loop, "sse-client-loop");    t.setDaemon(true);    t.start();  }  private void loop() {    while (running) {      try {        HttpRequest req = HttpRequest.newBuilder(URI.create(url))            .header("Accept", "text/event-stream")            .timeout(Duration.ofSeconds(30)). GET().build();        HttpResponse<InputStream> resp = http.send(req, HttpResponse.BodyHandlers.ofInputStream());        if (resp.statusCode() != 200) {          sleep(1500);          continue;        }        try (var is = resp.body();             var br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8))) {          String line;          String event = "message";          StringBuilder data = new StringBuilder();          while ((line = br.readLine()) != null) {            if (line.isEmpty()) { // Complete event              if (!data.isEmpty()) {                fire(event, data.toString());                data.setLength(0);                event = "message";              }              continue;            }            if (line.startsWith(":")) continue; Comment            if (line.startsWith("event:")) {              event = line.substring("event:".length()).trim();            } else if (line.startsWith("data:")) {              if (!data.isEmpty()) data.append('\n');              data.append(line.substring("data:".length()).trim());            }          }        }      } catch (Exception e) {        sleep(1500);      }    }  }  private void fire(String type, String data) {    for (listener l : listeners) {      try {        l.onEvent(type, data);      } catch (Exception ignored) {      }    }  }  public interface Listener {    void onEvent(String type, String data);  }}

8.5 UiBroadcaster

TheUiBroadcasteris a small helper class that distributes UI updates thread-safely to several active UIs. It maintains a list of currently connectedUI instances and executes provided commands within the respective UIcontext (ui.access(...)). This allows reactions to external signals (SSE) to be safely integrated into the VaadinUI without risking UIThread‑violations. The view registers on the attach and deregisters on the detach, thus avoiding zombie‑references.

xmlCopy

package com.svenruppert.flow.views.dashboard;//--- Simple broadcaster to distribute UI updates thread-safe ---public final class UiBroadcaster {  private UiBroadcaster() {  }  private static final List<UI> UI_LIST = new CopyOnWriteArrayList<>();  public static void register(UI ui) {    UI_LIST.add(ui);  }  public static void unregister(UI ui) {    UI_LIST.remove(ui);  }  public static void broadcast(Command task) {    for (UI ui : List.copyOf(UI_LIST)) {      try {        ui.access(task);      } catch (Exception ignored) {      }    }  }}

9. Summary

9.1 Evaluation of the “Signal-per-SSE, Data-per-REST” pattern

The demonstration showed that the separation of signalling and data transmission enables a clear and comprehensible architecture. SSE is ideal for informing clients of changes in real time, while REST takes care of the reliable and flexible delivery of the actual data. This division of labour results in low communication costs and, at the same time, a high level of transparency for users.

9.2 Didactic benefits and reusability

The scenario presented is deliberately kept simple to make the functionality of SSE in interaction with a Vaadin Flow application understandable. The integration of a CLI for data entry makes it easier to understand the chain of events, signals and retrieval clearly. This makes the example suitable not only for technical experiments, but also for training, workshops and teaching materials. Due to its modular structure, it can be easily extended or integrated into more complex projects, for example, as a basis for further experiments in the context of the URL shortener open source project.

]]>Uncategorizedhttps://svenruppert.com/posts/how-and-why-to-use-the-classic-observer-pattern-in-vaadin-flow/Mon, 01 Sep 2025 11:32:18 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/how-and-why-to-use-the-classic-observer-pattern-in-vaadin-flow/1. Introduction and motivation The observer pattern is one of the basic design patterns of software development and is traditionally used to decouple state changes and process them. Its origins lie in the development of graphical user interfaces, where a shift in the data model required synchronising several views immediately, without a direct link between these views. This pattern quickly established itself as the standard solution to promote loosely coupled architectures.<![CDATA[

1. Introduction and motivation

The observer pattern is one of the basic design patterns of software development and is traditionally used to decouple state changes and process them. Its origins lie in the development of graphical user interfaces, where a shift in the data model required synchronising several views immediately, without a direct link between these views. This pattern quickly established itself as the standard solution to promote loosely coupled architectures.

1. 1. Introduction and motivation
2. 2. The classic observer pattern
3. 3. Observer Pattern in Vaadin Flow
4. 4. Differences between classic Observer and Vaadin Flow
5. 5. When does the classic observer pattern still make sense?
6. 6. Example: Combination of Observer Pattern and Vaadin Flow (with external producer)
7. 7. Best Practices and Pitfalls
8. 8. Conclusion

At its core, the Observer Pattern addresses the challenge that events or state changes do not remain isolated; instead, they must be processed by different components. This creates an asynchronous notification model: The subject informs all registered observers as soon as its state changes. The observers react to this individually, without the subject having to know their concrete implementations.

The source code with the examples can be found in the following git-repo:https://github.com/Java-Publications/Blog—Vaadin—How-to-use-the-Observer-Pattern-and-why** __**

In modern software architectures, characterised by interactivity, parallelism, and distributed systems, this basic principle remains highly relevant. It continues to be used in the field of UI development, as user interactions and data flows must be handled dynamically and in a reactive manner. Frameworks such as Vaadin Flow take up this idea, but go beyond the classic implementation by providing mechanisms for state matching, server-side synchronisation and client-side updates.

The motivation to apply the Observer Pattern in the context of Vaadin Flow lies in comparing proven yet simple patterns and their further development within a modern UI framework. This not only sharpens the understanding of loose coupling and event handling, but also creates a practical reference to today’s web applications.

2. The classic observer pattern

The classical observer pattern describes the relationship between a subject and a set of observers. The subject maintains the current state and automatically informs all registered observers of any relevant change. This achieves a loose coupling: the subject only knows the interface of its observers, but not their concrete implementation or functionality.

In its original form, according to the “Gang of Four” definition, the pattern consists of two central roles. The subject manages the list of its observers and provides methods of logging in and out. As soon as the internal state changes, it calls a notification method for all observers. The observers implement an interface through which they are informed and can define their own reaction. This allows state changes to be propagated without creating complex coupling or cyclic dependencies.

To avoid outdated dependencies and to maintain control over the API, an independent, generic variant with its own interfaces is shown below. The aim is to demonstrate how it works in a JUnit 5 test without the main method. The output is a logger viaHasLogger from the packagecom.svenruppert.dependencies.core.logger.

xmlCopy

package com.svenruppert.demo;import com.svenruppert.dependencies.core.logger.HasLogger;import org.junit.jupiter.api.Test;import java.util.List;import java.util.concurrent.CopyOnWriteArrayList;import static org.junit.jupiter.api.Assertions.assertEquals;public class ObserverDemoTest { @Test void beobachter_erhaelt_updates_und_letzten_wert() { var sensor = new TemperatureSensor(); var anzeigeA = new TemperaturAnzeige(); var displayB = new Temperature display(); sensor.addObserver(displayA); sensor.addObserver(displayB); sensor.setTemperature(22); assertEquals(22, displayA.lastObservedTemp); assertEquals(22, sensor.getTemperature()); sensor.setTemperature(25); assertEquals(25, sensor.getTemperature()); assertEquals(25, displayA.lastObservedTemp); assertEquals(25, displayB.lastObservedTemp); sensor.removeObserver(displayA); sensor.setTemperature(21); assertEquals(21, sensor.getTemperature()); assertEquals(25, displayA.lastObservedTemp); assertEquals(21, displayB.lastObservedTemp); } /** * Minimalist, own Observer API (generic, thread-safe implemented in the subject). */ interface Observer<T> { void onUpdate(T value); } interface Subject<T> { void addObserver(Observer<T> o); void removeObserver(Observer<T> o); void notifyObservers(T value); } /** * Concrete subject: maintains a temperature and informs observers of changes. */ static class Temperature Sensor implements Subject<Integer> { private final List<Observer<Integer>> observers = new CopyOnWriteArrayList<>(); Private int temperature; @Override public void addObserver(Observer<Integer> o) { observers.add(o); } @Override public void removeObserver(Observer<Integer> o) { observers.remove(o); } @Override public void notifyObservers(Integer value) { observers.forEach(o -> o.onUpdate(value)); } public int getTemperatur() { return temperatur; } public void setTemperature(int newtemperature) { this.temperature = newtemperature; notifyObservers(temperature); } } /** * Concrete Observer: remembers the last value and logs it. */ static class temperature display implements Observer<Integer>, HasLogger { Integer lastObservedTemp; @Override public void onUpdate(Integer value) { lastObservedTemp = value; logger().info("New temperature: {}°C", value); } }}

In the example, the company’s own API defines two lean interfaces: Subject manages observers and propagates changes, while Observer reacts to updates. The implementation of the Subject uses a CopyOnWriteArrayList to handle concurrent logins and logoffs during notification robustly. The JUnit5Test demonstrates the expected behaviour: By setting the temperature twice, notifications are triggered twice. The temperature display logs the values via HasLogger and retains the last observed value for assertion. This demonstrates the basic semantics of the Observer Pattern – loose coupling, clear responsibilities and event-driven updating – in a modern, framework-free Java variant.

3. Observer Pattern in Vaadin Flow

While the classic Observer Pattern has its origins in desktop programming, Vaadin Flow elevates the concept to a higher level of abstraction. Since Vaadin operates on the server side and manages the UI state centrally, synchronisation between the data model, user interface, and user interactions is an integral part of the framework. Events and listeners take on the role of the observer mechanism.

A central element is theValueChangeListener , which is automatically triggered when changes are made to input components such as text fields or checkboxes. Here, the UI component acts as a subject, while listeners represent the observer. As soon as the user makes an input, an event is generated, and all registered listeners are informed. All the developer has to do is implement the desired response without worrying about chaining event flows.

Additionally, Vaadin provides a powerful tool with theBinder , which enables bidirectional data binding between the data model and UI components. Changes to a field in the form are automatically propagated to the bound object, while changes in the object are immediately visible in the UI elements. The binder thus adopts a bidirectional observation that goes beyond the possibilities of the classic observer pattern.

Another example of the observer pattern’s implementation in Vaadin Flow is theEventBus mechanism , which enables components or services to exchange information about events in a loosely coupled manner. Here, it becomes clear that Vaadin adopts the basic idea of the pattern, but expands it in the context of a web application to include aspects such as thread security, lifecycle management, and client-server synchronisation.

The strength of Vaadin Flow lies in the fact that developers no longer have to implement observer interfaces explicitly; instead, they can establish observation relationships via declarative APIs, such as addValueChangeListener, addClickListener, or via binder bindings. This does not abolish the pattern, but integrates it deeply into the architecture and adapts it to the special requirements of a server-side web UI.

Vaadin Flow implements the basic idea of the Observer Pattern along two levels. At the UI level, component-specific events (e.g., ValueChangeEvent, ClickEvent) exist that are registered via listeners and trigger changes to the representation or state model. This mechanism corresponds to a finely granulated, component-internal observer and is conveyed via an internal EventBus for each component. At the application level, a domain-level observation structure is also recommended to model state changes outside the UI and to mirror them in the UI in a decoupled manner. This allows long-lived domain objects and short-lived UII punches to be cleanly separated.

A minimal example exclusively with Vaadin on-board tools shows the observation of UIC conditions as well as the binding to a model via binder:

javaCopy

importcom.svenruppert.flow.MainLayout;importcom.vaadin.flow.component.button.Button;importcom.vaadin.flow.component.html.Div;importcom.vaadin.flow.component.orderedlayout.VerticalLayout;importcom.vaadin.flow.component.textfield.TextField;importcom.vaadin.flow.data.binder.binder;importcom.vaadin.flow.router.Route;@Route(value="observer-bordmittel",layout=MainLayout.class)publicclassOn-boardResourcesViewextendsVerticalLayout{publicBordmittelView(){varname=newTextField("Name");varlog=newDiv();varsave=newButton("Save");//(1) Component-internal observer: reacts to ValueChangeEventname.addValueChangeListener(e->log.setText("UI → UI: Name changed to '"+e.getValue()+"'"));//(2) Binder as a bidirectional observation between UI and modelvarbinder=newBinder<>(Person.class);varmodel=newPerson();binder.forField(name).bind(Person::getName,Person::setName);binder.setBean(model);// (3) Action: reads current model state (powered by the binder)store.addClickListener(e->log.setText("Model → Action: saved name = "+model.getName()));add(name,store,log);}staticclassPerson{privateStringname;publicStringgetName(){returnname;}publicvoidsetName(Stringname){this.name=name;}}}

Here, addValueChangeListener(…), the role of the observer at the component level, while the binder synchronises the changes between the input field and the domain object. Without additional infrastructure, this creates a transparent, declarative observation chain from the UI to the model and back to the UI. It is important to note that the button does not trigger a new save, but reads the model state that has already been synchronised by the binder and makes it visible. It thus serves as an action-level observer that confirms and logs the current consistency between UI and model.

4. Differences between classic Observer and Vaadin Flow

The comparison between the classic Observer Pattern and its implementation in Vaadin Flow reveals that both approaches share the same basic idea, but operate at different levels of abstraction. While the classic pattern relies on purely object-oriented structures – subject and observer communicate directly via defined interfaces – Vaadin Flow shifts observation to a framework-supported environment with integrated event control.

A significant difference lies inlife cycle management. In the classic observer, subjects and observers remain the responsibility of the developer; Registration and deregistration must be done manually, otherwise there is a risk of memory leaks. Vaadin Flow integrates observation into the lifecycle of UI components. Listeners are registered when a component is created and automatically resolved when the component is removed, reducing administrative overhead.

ThreadSecurity also differs significantly. The classic observer pattern works in a local context and has no UI thread binding. Vaadin Flow, on the other hand, must guarantee that UI updates are done exclusively in the UI thread. This forces the use of mechanisms such as UI.access(…)as soon as events from foreign threads are processed.

Another difference concerns thedirection and range of the synchronisation. In the classic model, notifications are limited to the objects involved. Vaadin Flow extends this principle to client-server communication: changes to the UI are automatically synchronised to the server, while server-side state changes are, in turn, made visible in the UI on the client side.

Finally, thegranularity of the observation varies. Classic observers usually react to gross changes in state. Vaadin Flow, on the other hand, offers a variety of specialised event types (e.g. ValueChange, Click, Attach/Detach) that allow for very finely tuned reactions. In combination with the binder , a bidirectional coupling between model and UI is created that goes far beyond the original idea of the pattern.

In summary, the classic observer pattern provides the theoretical basis. Still, Vaadin Flow abstracts and expands it by integrating lifecycle, thread security, and client-server synchronisation, thereby enabling a higher degree of robustness and productivity.

5. When does the classic observer pattern still make sense?

Although Vaadin Flow already integrates many observation mechanisms, there are scenarios in which the classic observer pattern can also be used sensibly in modern Vaadin applications. The crucial point here is theseparation between UI logic and domain logic. While listeners and binders in Vaadin handle the synchronisation between user input and UI components, events often arise in the business layer that must be managed independently of the UI.

A typical field of application isinternal domain events. For example, a background process can receive a new message, complete a calculation, or import data from an external system. These events should not be directly bound to UI components; instead, they should be distributed throughout the system. Here, the classic observer pattern is suitable for informing interested services or components of such changes without creating a direct link.

The pattern also plays a role in the context ofintegrations with external systems. When a REST service or messaging infrastructure feeds events into the system, the observer pattern can be used to propagate those events within the domain first. Only in the second step is it decided whether and how this information is passed on to a Vaadin UI. This keeps the dependency on the UI low, and the domain layer remains usable outside of a UI context.

Another reason for using the classic observer pattern is testabilityand reusability. Unit testing at the domain layer is easier to perform when observation mechanisms do not depend on Vaadin-specific APIs. This allows complex interactions to be tested in isolation before they are later wired to the UI.

In summary, the classic observer pattern complements the mechanisms available in Vaadin Flow for UI-independent, domain-internal events or integrations with external sources. It enables loose coupling, increases testability and ensures clear demarcations between layers – a principle that is a decisive advantage, especially in more complex applications.

6. Example: Combination of Observer Pattern and Vaadin Flow (with external producer)

An efficient variant arises when messages are not generated by the UI itself, but come from an external producer outside the UI life cycle. Typical sources are background processes, REST integrations or message queues. The Observer Pattern ensures that these events can be distributed to all interested components, regardless of the UI.

The following example extends the previous construction: A MessageProducer periodically generates messages in a separate thread and passes them to the MessageService. The Vaadin view remains a pure observer, reflecting only the messages in the UI.

xmlCopy

//Bootstrap: starts the producer on startupfinal class ApplicationBootstrap { private static final MessageService SERVICE = new MessageService(); static { MessageProducer.start(SERVICE); } private ApplicationBootstrap() { } static MessageService messageService() { return SERVICE; }}//External producer that periodically generates newspublic class MessageProducer { private static ScheduledExecutorService scheduler; private MessageProducer() { } static void start(MessageService service) { scheduler = Executors.newSingleThreadScheduledExecutor(); scheduler.scheduleAtFixedRate( new CronJob(service), 0, 2, SECONDS); } static void stop() { if (scheduler != null) { scheduler.shutdownNow(); } } public record CronJob(MessageService service) implements Runnable, HasLogger { @Override public void run() { logger().info("Next message will be sent.."); service.newMessage("Tick @ " + Instant.now()); } }}//Domain Servicepublic class MessageService implements Subject<String> , HasLogger { private final List<Observer<String>> observers = new CopyOnWriteArrayList<>(); @Override public void addObserver(Observer<String> o) { logger().info("New Observer added: {}", o); observers.add(o); } @Override public void removeObserver(Observer<String> o) { logger().info("Observer removed: {}", o); observers.remove(o); } @Override public void notifyObservers(String value) { logger().info("Notifying {} observers with the value {} ", observers.size(), value); observers.forEach(o -> o.onUpdate(value)); } public void newMessage(String msg) { notifyObservers(msg); }}public interface Observer<T> { void onUpdate(T value);}public interface Subject<T> { void addObserver(Observer<T> o); void removeObserver(Observer<T> o); void notifyObservers(T value);}

This structure creates a complete message flowproducer → domain → UI. The domain layer is independent of Vaadin and can be tested in isolation. The UI remains limited to the display and updates itself exclusively in the UI thread.

The new messages appear every two seconds, as the MessageProducer is configured in the background with scheduleAtFixedRate. When it starts, it immediately generates a message and then periodically generates a new one every two seconds. These are distributed to the registered observers via the MessageService.Important: For server-side changes to be reflected in the browser without user interaction, ServerPush or Polling must be enabled. In this example, @Push on the View ensures that ui.access(…) actively sends the changes to the client. Alternatively, polling (e.g. @Poll(2000) or UI#setPollInterval). The view receives the events and sets the text in the div using UI.access(…) in UI Thread, so that the most recent message is visible.

This uses the Observer Pattern to integrate external events into a Vaadin Flow application reliably.

7. Best Practices and Pitfalls

The integration of the classic Observer Pattern into a Vaadin Flow application brings both advantages and specific challenges. To ensure a robust and maintainable architecture, several best practices should be considered.

Observer lifecycle management. A common mistake is to leave Observer permanently registered in the service, even if the associated UI component has already been destroyed. This leads to memory leaks and unexpected calls to views that no longer exist. Therefore, observers in Vaadin views should always be registered inonAttach and removed in onDetach. In this way, the observation remains clearly linked to the life cycle of the UI.

Use of suitable data structures
Instead of simple lists, it is recommended to use concurrent, duplicate-free data structures, such as ConcurrentHashMap or CopyOnWriteArraySet, for managing observers. This avoids duplicate registrations and ensures thread safety, even if multiple threads add or remove observers at the same time.

Thread security and UI updates. Since external events are often generated in separate threads, it is imperative to install UI updates in Vaadin viaUI.access(…) be performed. Otherwise, there is a risk of race conditions and exceptions, because the UI components may only be changed from within the UI thread. Additionally, server push or polling should be enabled to ensure that changes are sent to the client without user interaction.

Domain and UI demarcation. The domain layer should remain completely free of Vaadin-specific dependencies. This ensures that business logic can be tested and reused outside of the UI context. The UI only registers itself as an observer and takes over the representation of the events delivered by the domain.

Error handling and logging
Events should be handled robustly to prevent individual errors from interrupting the entire notification flow. Clean logging (e.g. via a common HasLogger interface) facilitates the analysis and traceability of the event flow.

Summary. The main pitfalls lie in improper lifecycle management, a lack of thread security, and overly tight coupling between the UI and domain. When these aspects are taken into account, the combination of the Observer Pattern and Vaadin Flow enables a clear separation of responsibilities, testable business logic, and a reactive, user-friendly interface.

8. Conclusion

A look at the Observer Pattern in combination with Vaadin Flow reveals that it remains a relevant and compelling design pattern, whose benefits extend beyond classic desktop or console applications. In its original form, it provides apparent decoupling of state changes and their processing, thus creating a clean basis for loosely coupled architectures.

Vaadin Flow abstracts and extends this approach by integrating observation mechanisms directly into the lifecycle of UI components, providing a variety of specialised events, and automating synchronisation between server and client. This relieves the developer of many tasks that could still be solved manually in the classic observer pattern, such as memory management or thread security.

Nevertheless, the classic observer pattern remains essential in areas where UI-independent events occur or external systems are connected. The combination of both approaches – a clearly defined domain layer with observers and a Vaadin UI based on it – creates an architecture that is both loosely coupled, testable, and extensible.

Overall, it can be said that the Observer Pattern forms the theoretical basis, while Vaadin Flow provides the practical implementation for modern web applications. Those who consciously combine both approaches benefit from robust and flexible systems that are prepared for future requirements.

]]>Design PatternJavaVaadinhttps://svenruppert.com/posts/password-security-why-hashing-is-essential/Fri, 29 Aug 2025 14:53:27 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/password-security-why-hashing-is-essential/Password security is an often underestimated but critical topic in software development. Databases containing millions of user logins are repeatedly compromised - and shockingly, often, it turns out that passwords have been stored in plain text. This gives attackers direct access to sensitive account data and opens the door to identity theft, account takeovers and other attacks.<![CDATA[

Password security is an often underestimated but critical topic in software development. Databases containing millions of user logins are repeatedly compromised - and shockingly, often, it turns out that passwords have been stored in plain text. This gives attackers direct access to sensitive account data and opens the door to identity theft, account takeovers and other attacks.

In this blog post, we discuss why passwords should always be stored hashed, the attack methods available, and how you can implement an initial secure implementation with Java in your application. We also examine the differences between PBKDF2, BCrypt, and Argon2 and explain best practices for handling passwords in software development.

1. Passwords and hashing
2. What are brute force and rainbow table attacks on passwords?
3. And how do I do this now in my application?
4. How safe is PBKDF2?
5. Using stronger hashing algorithms like Argon2
6. Application of Argon2 in Java
7. Generate the SALT value.
8. What is the procedure for checking the username-password combination during the login process?
9. A few more words about strings in Java
   1. So what can you do about it now?
10. Conclusion

Passwords and hashing

Passwords should never be stored in plain text but should always be hashed to ensure the security of user data and prevent misuse. If a database is compromised and passwords are only stored in plain text, attackers have direct access to users’ sensitive credentials. This can have serious consequences, as many people use the same password for multiple services. Hashing passwords significantly reduces this risk because attackers only see the hash values ​​, not the actual ones.

A key advantage of hashing is its one-way function. A hash value can be generated from a password, but inferring the original password is virtually impossible. This makes it extremely difficult to misuse the access data resulting from a data leak. This protection mechanism applies not only to external attacks but also to internal security risks. If passwords are stored in plain text, employees with access to the database, for example, could view and misuse this information. Hashed passwords largely eliminate such insider threats.

In addition, compliance with legal requirements and security standards such as the General Data Protection Regulation (GDPR) or the Payment Card Industry Data Security Standard (PCI-DSS) requires the protection of passwords. Hashing passwords is important to meet these standards and avoid legal consequences.

What are brute force and rainbow table attacks on passwords?

Brute force attacks and rainbow table attacks are two methods attackers use to decrypt passwords or other secrets. ABrute force attack works by systematically trying every possible password combination until finding the right one. We start with the simplest combinations, such as “aaa” or “1234”, and check every possible variant. Although brute force attacks can theoretically always be successful, their effectiveness depends heavily on the complexity of the password and computing power. A short password can be cracked in a few seconds, while a long and complex password increases the effort significantly. Measures such as longer passwords, limiting the number of login attempts per unit of time (e.g. blocking after several failed attempts) and the use of hashing algorithms with many iterations make brute force attacks significantly more difficult and time-consuming.

In contrast, theRainbow table attacks pre-built tables that contain many hash values ​​and their associated plaintext passwords. Attackers compare the stored hash of a password with the hashes in the table to find the original password. This method is significantly faster than brute force because the passwords must not be re-hashed every time. However, rainbow tables only work if the hashing method does not use a salt value. A salt value is a random value added to each password hash, so even identical passwords produce different hashes. Without salt, attackers could use the same table across many other systems, but this is no longer possible with salt.

Using modern hashing algorithms such as PBKDF2, BCrypt, or Argon2 is crucial to protecting yourself from both attack methods. These algorithms combine salts and multiple iterations to significantly increase attacker effort. Additionally, long and complex passwords make brute-force attacks virtually impossible as the number of possible combinations increases exponentially. In summary, combining strong passwords, salts, and secure hashing algorithms effectively defends against brute force and rainbow table attacks.

And how do I do this now in my application?

To securely hash a password in Java, it is recommended to use a specialised hash function likePBKDF2 ,BCrypt , or Argon2. These algorithms are specifically designed to make attacks such as brute force or rainbow table attacks more difficult. One way to implement this with the Java standard library is to usePBKDF2.

First, aSalt is generated, a random sequence of bytes generated for each password individually to ensure that two identical passwords have different hashes. The class SecureRandom can be used to create a 16-byte salt. Then, with the class PBEKeySpec, a key is generated based on the password, the salt, the desired number of iterations (e.g. 65,536) and the key length (e.g. 256 bits). With the help of SecretKeyFactory and the algorithm specification “PBKDF2WithHmacSHA256” the password hash is created. The resulting hash is finally encoded in Base64 to store it in a readable form.

The hashed password is often stored along with the salt, separated by a colon (:). This makes it easier to verify later by extracting the salt again and using it for the hash calculation. In addition, there are also external libraries such asSpring Security orBouncyCastle , which allow easier integration of algorithms likeBCrypt orArgon2 make possible. These often offer even more security and flexibility.

javaCopy

importjava.security.NoSuchAlgorithmException;importjava.security.spec.InvalidKeySpecException;importjava.security.spec.KeySpec;importjava.util.Base64;importjavax.crypto.SecretKeyFactory;importjavax.crypto.spec.PBEKeySpec;publicclassPasswordHasher{privatestaticfinalintITERATIONS=65536;privatestaticfinalintKEY_LENGTH=256;privatestaticfinalStringALGORITHM="PBKDF2WithHmacSHA256";publicstaticStringhashPassword(Stringpassword,Stringsalt){try{KeySpecspec=newPBEKeySpec(password.toCharArray(),salt.getBytes(),ITERATIONS,KEY_LENGTH);SecretKeyFactoryfactory=SecretKeyFactory.getInstance(ALGORITHM);byte[]hash=factory.generateSecret(spec).getEncoded();returnBase64.getEncoder().encodeToString(hash);}catch(NoSuchAlgorithmException|InvalidKeySpecExceptione){thrownewRuntimeException("Error while hashing password",e);}}publicstaticvoidmain(String[]args){Stringpassword="myPassword";Stringsalt="randomSalt";StringhashedPassword=hashPassword(password,salt);System.out.println("hasehd password: "+hashedPassword);}}

However, there are a few points here that you should take a closer look at.

How safe is PBKDF2?

PBKDF2 is a proven and secure password hashing algorithm, but its security depends heavily on the correct implementation and parameters used. Various factors influence the safety of PBKDF2. A key aspect is the number of iterations, representing the work factor. The higher the number of iterations, the more computing power is required for hashing. Experts recommend at least 100,000 iterations, although even higher values ​​are often chosen for modern applications to make brute-force attacks more difficult. Another crucial factor is the salt value, a random and unique value that is regenerated for each password. In addition, PBKDF2 can be configured with a variable key length, for example, 256 bits, which increases security. Modern applications often use hash functions such as SHA-256 or SHA-512.

PBKDF2 has several strengths. The algorithm is proven and has been used for years in security-critical applications such as WPA2 and encrypted storage systems. It is easy to implement as extensive support in many programming languages ​​and libraries is based on a recognised standard defined in RFC 8018. Nevertheless, PBKDF2 also has weaknesses. The algorithm is not optimised explicitly against GPU or ASIC-based attacks, allowing attackers with specialised hardware to efficiently carry out brute force attacks. Compared to modern algorithms such as Argon2, PBKDF2 requires a higher number of iterations to achieve similar levels of security. In addition, the further development of PBKDF2 is progressing more slowly, which means it is considered less adapted to current threats than modern alternatives such as Argon2.

However, PBKDF2 can be used safely if a few key conditions are met: a unique salt value, the iteration count should be at least 100,000 (ideally more), and a modern hash function such as SHA-256 should be used. In addition, strong password guidelines, such as a sufficient minimum length, should complement security.

Argon2 is increasingly recommended as an alternative to PBKDF2. Argon2, the Password Hashing Competition (PHC) winner in 2015, is more modern and better adapted to current threats. Its memory intensity provides better protection against GPU and ASIC attacks and offers flexible configuration options regarding work factor, memory requirements and parallelism.

In summary, PBKDF2 is secure when implemented correctly but has weaknesses compared to specialised hardware. Argon2 is preferable for new applications because it is better suited to modern security requirements.

Using stronger hashing algorithms like Argon2

Argon2 is a modern algorithm for secure password hashing and was developed in 2015 as part of thePassword Hashing Competition (PHC). It is considered one of the most secure approaches to storing passwords today and, as already mentioned, offers adequate protection against brute force attacks as well as attacks using GPUs or specialised hardware solutions such as ASICs. This is achieved due to the fact that Argon2 is both memory and compute-intensive, forcing attackers with parallel hardware to expend significant resources.

Argon2 exists in three variants, each optimised for different use cases. The first variant,Argon2i , is designed to be particularly memory-safe. It performs memory-intensive operations independently of the input data and is resistant to side-channel attacks such as timing attacks. This makes Argon2i ideal for applications where data protection is a top priority.

The second variant,Argon2d , is specifically optimised for attack resistance. It works data-dependently, which makes it particularly robust against GPU-based attacks. However, Argon2d is more vulnerable to side-channel attacks and is, therefore, less suitable for securing sensitive data.

The third variant, Argon2id, offers a balanced combination of both approaches. It starts with a memory-safe approach, like that used by Argon2i, and then moves to a data-dependent process that ensures attack resistance. This mix makes Argon2id the preferred choice for most use cases, as it combines data protection and attack resilience.

One of Argon2’s key strengths is its customizability. The algorithm allows developers to configure three primary parameters: memory consumption, computational cost and parallelism. Memory consumption defines how much memory is used during the hashing operation and makes parallel hardware attacks expensive. Computational cost indicates the number of iterations the algorithm goes through, while parallelism determines the number of threads working simultaneously. By adjusting these parameters, the algorithm can be tailored to the specific requirements of an application or the available hardware.

Another advantage of Argon2 is its resistance to modern attacks. The combination of memory and computing-intensive processes makes it difficult for attackers to crack passwords using brute force or specialised hardware. This makes Argon2 ideal for use in safety-critical applications.

Argon2 is used in many modern cryptography and password management libraries. Developers can implement the algorithm in various programming languages ​​, such as Java, Python, or C.

In summary, Argon2 is one of the safest password-hashing algorithms. The variantArgon2id is especially recommended as a standard for new projects because it provides protection against both side-channel attacks and high attack resistance.

Application of Argon2 in Java

OneArgon2 inJava To use it, you can use libraries likeJargon2 orBouncy Castle because Java does not natively support Argon2. WithJargon2 , Argon2 is very easy to integrate and use. To do this, first, add the following Maven dependency:

xmlCopy

<dependency><groupId>com.kosprov</groupId><artifactId>jargon2-api</artifactId><version>Latest Version Number</version></dependency>

The code to create a hash and verify a password might look like this:

javaCopy

importcom.kosprov.jargon2.api.Jargon2;publicclassArgon2Example{publicstaticvoidmain(String[]args){Stringpassword="DeinSicheresPasswort";Jargon2.Hasherhasher=Jargon2.jargon2Hasher().type(Jargon2.Type.ARGON2id).memoryCost(65536).timeCost(3).parallelism(4).saltLength(16).hashLength(32);StringhashedPassword=hasher.password(password.getBytes()).encodedHash();System.out.println("hashed password: "+hashedPassword);booleanmatches=Jargon2.jargon2Verifier().hash(hashedPassword).password(password.getBytes()).verifyEncoded();System.out.println("password correct: "+matches);}}

If you want to use a more comprehensive cryptography library, you canuse Bouncy Castle. The corresponding Maven dependency is:

xmlCopy

<dependency><groupId>org.bouncycastle</groupId><artifactId>bcprov-jdk18on</artifactId><version>Latest Version Number</version></dependency>

An example of using Argon2 with Bouncy Castle looks like this:

javaCopy

importorg.bouncycastle.crypto.params.Argon2Parameters;importorg.bouncycastle.crypto.generators.Argon2BytesGenerator;importjava.security.SecureRandom;publicclassArgon2BouncyCastleExample{publicstaticvoidmain(String[]args){Stringpassword="securePassword";byte[]salt=newbyte[16];newSecureRandom().nextBytes(salt);Argon2Parameters.Builderbuilder=newArgon2Parameters.Builder(Argon2Parameters.ARGON2_id).withSalt(salt).withMemoryPowOfTwo(16).withParallelism(4).withIterations(3);Argon2BytesGeneratorgenerator=newArgon2BytesGenerator();generator.init(builder.build());byte[]hash=newbyte[32];generator.generateBytes(password.getBytes(),hash);System.out.println("hashed password: "+bytesToHex(hash));}privatestaticStringbytesToHex(byte[]bytes){StringBuilderhexString=newStringBuilder();for(byteb:bytes){hexString.append(String.format("%02x",b));}returnhexString.toString();}}

Jargon2 was developed specifically for Argon2 and is, therefore, easy to useBouncy Castle for more complex cryptographic requirements. It is recommended in most cases for the pure use of Argon2.

Generate the SALT value.

So far, it has always been mentioned that producing a good SALT value is important. But how can you do that?

We will examine the topic in detail in the next blog post. Unfortunately, at this point, it would go beyond the scope of this article.

In this blog post, we will examine a basic initial implementation. You will quickly be confronted with an implementation that could look like this.

javaCopy

publicStringgenerateSalt(intlength){SecureRandomrandom=newSecureRandom();byte[]salt=newbyte[length];random.nextBytes(salt);returnBase64.getEncoder().encodeToString(salt);}

However, there are some minor comments here.

Basically, creating a salt with a newly instantiated SecureRandom is not fundamentally wrong in any method, but it increases the chancethat the same seeds are used at very short intervals or under certain circumstances. In practice, that israrely a problem; however, it is good practice to have a single (static) instance of SecureRandom to be used per application (or per class).

Why?

• SecureRandom gets its seed (among other things) from the operating system (e.g /dev/urandom in Linux).
• Each re-creation of one SecureRandom can lead to unnecessary system load and, theoretically, minimal increased risk of repetitions.
• At just one SecureRandominstance, a pseudo-random number generator is continued internally, making duplicates very unlikely.

javaCopy

importjava.security.SecureRandom;importjava.util.Base64;publicclassSaltGenerator{privatestaticfinalSecureRandomRANDOM=newSecureRandom();publicstaticStringgenerateSalt(intlength){byte[]salt=newbyte[length];RANDOM.nextBytes(salt);returnBase64.getEncoder().encodeToString(salt);}}

This way, (a) is not repeated every time it is called SecureRandom, and (b) reduces the risk of randomly generating identical salts. Of course, purely statistically speaking, collisions can theoretically still occur, but the probability is negligible if the salt length is long enough.

However, this is only the beginning of the discussion; more on that in the second part.

What is the procedure for checking the username-password combination during the login process?

A standardised procedure is followed to check the combination of user name and password during a login process, ensuring that the data is processed securely. First, the user enters their login details via a form transmitted over HTTPS. The server then looks up the username in the database to retrieve relevant information such as password hash and salt. Care should be taken not to reveal whether a user exists.

The stored password hash is then compared with a recalculated hash of the entered password using the stored salt. If the values ​​match, the login is considered successful; otherwise, a generic error message is returned so as not to reveal additional information.

After successful verification, a session or JSON Web Token (JWT) is created to maintain the user’s authentication. The token does not contain any sensitive information, such as passwords.

A few more words about strings in Java

If we imagine that a password is in a text field of a (web) application, we will receive this password as a string. Yes, some will say, but what could be bad about that?

To do this, we need to examine briefly how strings are handled in the JVM and where attack vectors can lie.

In Java, strings are immutable, which means that once created, String-Object can no longer be changed. When a string is manipulated through concatenation, a new string object is created in memory while the old one continues to exist until it is removed by the garbage collector (GC). This behaviour can be problematic when sensitive data, such as passwords or cryptographic keys, is stored in strings. Because they cannot be overwritten directly, they may remain in memory longer than necessary and are potentially visible in a memory dump or readable by an attacker. Another problem is that the developer has no control over when the garbage collector removes the sensitive data. This can cause such data to remain in memory for an extended period and possibly become visible in logs or debugging tools.

So what can you do about it now?

A safer approach is to use this instead of strings char[] because they are changeable, and the memory can be specifically overwritten. This helps minimise the retention time of sensitive data, especially in security-critical applications where memory dumps or debugging tools could provide memory access.

The advantage of char[] lies in direct memory management: While the garbage collector decides itself when to remove objects, a char[]-Array is explicitly overwritten and thusimmediately deleted. This significantly reduces the risk of unauthorised access.

javaCopy

importjava.util.Arrays;publicclassSensitiveDataExample{publicstaticvoidmain(String[]args){char[]password={'s','e','c','r','e','t'};try{processPassword(password);}finally{// Überschreiben des Speichers mit Dummy-DatenArrays.fill(password,'\0');}}privatestaticvoidprocessPassword(char[]password){// Beispielhafte VerarbeitungSystem.out.println("Passwort verarbeitet: "+String.valueOf(password));}}

Next to char[], additional security measures should be taken, such as encrypting sensitive data and using SecretKeySpec from the package javax.crypto.spec. for cryptographic keys. This class allows handling cryptographic keys in byte arrays that can be overwritten after use, but I will write a separate blog post about that…

Conclusion

Secure storage of passwords is an essential part of every application to protect user data from misuse and unauthorised access. Plain text passwords pose a significant risk and should never be saved directly. Instead, modern hashing algorithms such as PBKDF2, BCrypt or Argon2 must be used to ensure security.

Brute force and rainbow table attacks illustrate the importance of salts and sufficiently complex hashing mechanisms. The risk of such attacks can be significantly reduced by using random salts and iterative hashing methods. Argon2id, in particular, offers high resistance to modern attack methods due to its storage and computing intensity and is recommended as the preferred solution.

In addition, passwords within the application should also be processed carefully. Using char[] instead of String to store sensitive data can help prevent unwanted memory leaks. It is also important not to store passwords unsecured in memory or in logs.

Secure password hashing procedures are a technical best practice and a legal requirement in many areas.

]]>JavaSecure Coding PracticesSecurityhttps://svenruppert.com/posts/what-makes-vaadin-components-special/Tue, 19 Aug 2025 13:09:52 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/what-makes-vaadin-components-special/From my experience, Vaadin has always stood out from other Java frameworks. Of course, it enables the creation of modern web UIs, but the real difference lies in its component architecture. This is not conceived as a short-term aid, but is consistently designed for maintainability and flexibility. It creates the possibility of running applications stably for many years while still being able to extend them step by step.<![CDATA[

From my experience, Vaadin has always stood out from other Java frameworks. Of course, it enables the creation of modern web UIs, but the real difference lies in itscomponent architecture. This is not conceived as a short-term aid, but is consistently designed for maintainability and flexibility. It creates the possibility of running applications stably for many years while still being able to extend them step by step.

Whether I use a component from the official library, develop my own, or integrate an npm package, Vaadin ensures that all building blocks follow a clear pattern. The connection between server and client remains consistent and reliable. This has allowed me to further develop applications over time without ever being forced to rebuild them entirely from scratch.

Note: This post builds on and reflects further upon the excellent original article by Sami Ekblad, who provided a concise overview of what makes Vaadin components special. My intention here is to expand on those ideas with reflections from my own long-term use in practice. -https://dev.to/samiekblad/what-makes-vaadin-components-special-9oo-temp-slug-1862339

---

More Than Just UI Widgets

In my practical work, I have observed that libraries such as React or Lit understand their components primarily as frontend building blocks. Vaadin deliberately takes a different approach. For me, the focus is onlongevity and extensibility , which are crucial in long-term projects. I have seen applications operated with Vaadin for more than a decade, and the underlying component architecture is designed precisely for this.

What I find particularly interesting is the possibility of implementing UI components either entirely server-side, purely client-side, or in a hybrid form.This freedom of choice is the real strength that sets Vaadin apart from traditional UI libraries and gives me the flexibility to choose the most appropriate solution for each use case.

---

Three Approaches to Vaadin Components

1. Server-Side Components

This is the most traditional route: development is done exclusively in Java. HTML templates or JavaScript are not necessary. With existing building blocks such as TextField, Button, or Grid, entire interfaces can be assembled. These can, in turn, evolve into higher-level components, such as an InvoiceEditor that directly integrates business logic and input validation.

Strengths:
Server-side components offer maximum control over the validation and security of user input, as all checks take place directly on the server and are thus protected against manipulation. At the same time, there is no dependency on frontend technologies, which reduces complexity and simplifies maintenance. Another advantage is the possibility of generating user interfaces dynamically at runtime, allowing flexible responses to changes in business logic or context. In addition, existing Java libraries and enterprise logic can be integrated seamlessly, creating a smooth transition between UI and backend.

Weaknesses:
Every user interaction requires a request to the server, which in scenarios with very high interactivity can lead to noticeable delays. In particular, with animations or complex drag-and-drop mechanisms, this approach reaches its limits, as such functions demand an immediate response in the browser that cannot be achieved without client-side logic.

---

2. Client-Server Components

Here, client-side web components (Lit, TypeScript, or Vanilla) are combined with a Vaadin Java API. This architecture unites a rich user experience with secure backend validation.

Example: A date picker with keyboard shortcuts, colour transitions and smooth animations, while still ensuring that inputs are reliably validated on the server.

Strengths:
Client-server components provide complete control over the DOM and all interactions in the browser. This makes it possible to realise complex and responsive user interfaces that keep pace with modern web standards. At the same time, business logic does not remain unprotected in the frontend but is validated and executed server-side in Java, ensuring a high level of security. Another benefit is the clear separation between UI and server logic, since communication is handled in a structured way by the framework, promoting clarity and maintainability.

Weaknesses:
A disadvantage is that developers need solid knowledge of both Java and TypeScript. This dual requirement can raise the entry barrier and lengthen the learning curve, since one must be familiar not only with server-side concepts but also with the particularities of client-side development. Especially in teams that have so far focused predominantly on one language, this can demand additional training and a more intensive onboarding process.

---

3. Integration of Third-Party Components

The modern web is rich in high-quality custom elements available via npm. With Vaadin, these elements can be easily integrated into an application. A small amount of TypeScript is sufficient to build the bridge to the Java world.

Strengths:
The integration of existing web components is particularly fast and straightforward, since ready-made elements from the npm ecosystem can be included with minimal effort. Developers benefit from the fact that only a small amount of boilerplate code is required to make a component usable. This leaves more scope for actual application logic. These integrated elements are also well-suited for visual enhancements or additional convenience features that may not be business-critical but still improve the user experience noticeably.

Weaknesses:
One drawback of integrating third-party components is that server-side validation is not automatically provided and must instead be implemented individually. Developers, therefore, need to add backend logic to ensure that inputs are reliably checked and possible manipulation is prevented. Another point concerns visual design: not all external components integrate seamlessly with the Vaadin design system. This can result in visual inconsistencies that impair a uniform appearance of the application and may require additional adaptation effort.

---

Making the Right Choice

The real question is not to commit permanently to a single approach. Each of the described variants brings specific strengths and weaknesses, which can be relevant in different contexts. In my projects, I have found the most outstanding value when combining the approaches. For example, I have been able to use server-side forms with clear validation logic alongside highly interactive client widgets, while also incorporating external components from the npm ecosystem. In this way, an application emerges in which different building blocks coexist harmoniously without losing their semantic meaning. At its core, a button remains a button – regardless of whether it is implemented server-side, client-side, or in a hybrid form.

---

Conclusion: Flexibility as a Guiding Principle

In my experience, Vaadin’s true strength does not lie solely in its library or the choice between Java and TypeScript, but in theflexibility that allows me as a developer to place logic exactly where it delivers the most significant benefit in terms of performance, security and maintainability. Particularly useful is the fact that existing components can be combined seamlessly with new technical approaches, enabling applications to grow continuously without requiring a fundamental redevelopment. This possibility of evolving systems organically has proved to be a decisive advantage in long-term projects.

For me, Vaadin components are therefore far more than simple UI building blocks. They embody an architectural principle designed for sustainability and future viability. This creates a stable foundation on which modern user interfaces can be developed that adapt flexibly to new requirements. In my work, I have repeatedly found that this approach allows new functionality to be introduced step by step without risking breaks in the existing architecture or being forced to restart a project from scratch.

---

In conclusion, what have your experiences been with the different approaches? Have you already worked with hybrid components that combine server-side and client-side logic, or have you instead found situations where a consistently pursued style has brought advantages? I am particularly interested in how you use these possibilities in practice, what challenges you have encountered, and which patterns have proved helpful for you.

]]>JavaVaadinhttps://svenruppert.com/posts/part-iii-webui-with-vaadin-flow-for-the-url-shortener/Fri, 15 Aug 2025 10:46:10 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/part-iii-webui-with-vaadin-flow-for-the-url-shortener/1. Introduction and objectives The first two parts of this series established the theoretical and practical foundations of a URL shortener in pure Java. We discussed the semantic classification of short URLs, the architecture of a robust mapping system, and the implementation of a REST-based service based on the JDK HTTP server. These efforts resulted in a functional, modularly extensible backend that creates, manages, and efficiently resolves short links. However, a crucial component was missing-a visual interface for direct user interaction with the system. This interface is essential for tasks such as manual link creation, viewing existing mappings, and analysing individual redirects.<![CDATA[

1. Introduction and objectives

The first two parts of this series established the theoretical and practical foundations of a URL shortener in pure Java. We discussed the semantic classification of short URLs, the architecture of a robust mapping system, and the implementation of a REST-based service based on the JDK HTTP server. These efforts resulted in a functional, modularly extensible backend that creates, manages, and efficiently resolves short links. However, a crucial component was missing-a visual interface for direct user interaction with the system. This interface is essential for tasks such as manual link creation, viewing existing mappings, and analysing individual redirects.

1. 1. Introduction and objectives
2. 2. UI strategy and technology choice
   1. 2.1 Why Vaadin Flow for a server-side interface?
   2. 2.2 UI paradigms for administration systems
   3. 2.3 Differentiation from client-side frameworks
3. 3. User interface architecture
   1. 3.1 Integration into the existing WAR structure
   2. 3.2 Separation of routing, views and services
   3. 3.3 Communication with the core module (DI without frameworks)
4. 4. User guidance and UX design
   1. 4.1 Navigation concept: Dashboard, Create, Overview
   2. 4.2 Input validation and feedback
   3. 4.3 Readability, design system and accessibility
5. 5. Implementation of the UI components with Vaadin Flow (including source code)
   1. 5.1 MainLayout: Navigation frame and routing
   2. 5.2 CreateView: Input form with validation
   3. 5.3 OverviewView: Grid-based management view
6. 5.4 Consistent behaviour for dialogues and notifications (with code)
   1. Success stories via Notification
   2. Error messages through field-based validation withBinder
   3. Destructive actions only after explicit confirmation
   4. Interaction and reuse
7. 7. Deploying the UI in the WAR context
   1. 7.1 Servlet configuration for Vaadin inweb.xml
   2. 7.2 Theme-Integration ohne Frontend-Toolchain
   3. 7.3 Operation in Tomcat or Jetty
8. 8. Conclusion and outlook

This is precisely where the third part of this series comes in. The focus is on the development of a graphical user interface based onVaadin Flow. The goal is to provide a web interface developed entirely in Java that eliminates the need for client-side frameworks and integrates seamlessly into the application’s existing WAR structure. The decision to use Vaadin Flow follows a pragmatic principle: Anyone who already writes Java should also be able to define interfaces in Java, with a type-safe layout, a declarative component model, and a clear separation between the UI and domain layers.

UI integration in our project serves not only to enhance functionality but also to achieve a didactic objective. In security-critical applications like URL shorteners, human interaction is key. A well-designed UI not only facilitates convenient operation but also prevents common errors and misuse through appropriate validation, clear feedback, and restrictive permissions concepts. This makes the user interface itself a security-relevant part of the architecture.

In the remainder of this article, we will examine the structure, design, and concrete implementation of the Vaadin interface in detail. This ranges from the module architecture to navigation and view components, including advanced features such as QR code generation and expiration time display. The goal remains to implement all functionality exclusively using Java’s built-in tools—in the spirit of a transparent, maintainable, and extensible system design.

The UI is not just an add-on, but a central component of a comprehensive shortener solution. It caters to both productive use and teaching and exploration, providing a sense of security and reliability.

2. UI strategy and technology choice

Choosing the right technology for the user interface of a technical service is far more than a matter of taste – it touches on key aspects such as maintainability, security model, development efficiency, and deployment strategy. In the context of our URL shortener, which deliberately avoids external frameworks and was implemented entirely in Java 24, it made sense to choose a solution for the UI that fits seamlessly into this philosophy. The decision was therefore made to useVaadin Flow , a server-side UI framework powered and developed entirely in Java.

2.1 Why Vaadin Flow for a server-side interface?

Vaadin Flow enables the development of component-based web applications in Java, without the need for JavaScript, front-end build processes, or manual maintenance of HTML, CSS, or TypeScript artefacts. All UI components, from simple input fields to complex tables, are described declaratively in Java, allowing the UI to seamlessly integrate into the existing modular structure. For a purely Java-based backend logic like our URL shortener, this results in a coherent overall architecture that eliminates the need for language or context switching, instilling confidence in its efficiency.

The logic is executed entirely server-side: User interactions trigger events that are processed on the server, with the presentation in the browser synchronised via the Vaadin client. This architecture not only simplifies state management but also reduces potential attack surfaces, as no server-side business logic migrates to the frontend.

2.2 UI paradigms for administration systems

When designing an admin interface, it’s not about aesthetic brilliance, but rather functional clarity, robust interactions, and data accountability. Users of a URL shortener—whether self-hosted, in educational institutions, or within an organisation—need an easy way to manually shorten new URLs, browse existing mappings, check expiration times, and delete potentially critical entries. These functions must be visible, traceable, and cancelable at any time.

Vaadin Flow offers a variety of ready-made components, such asGrid ,FormLayout ,Dialogue ,Notification orBinder that enable efficient implementation of such use cases – including validation, event control, and dynamic data binding. This not only saves development time but also enables a consistent user experience that scales elegantly with increasing complexity.

2.3 Differentiation from client-side frameworks

Unlike single-page application frameworks such as Angular, React or Vue, Vaadin Flow requiresno separate frontend build pipeline , no TypeScript toolchains, and no JSON serialisation of server communication. The entire application, including the UI, can be deployed as aclassic WAR Package and run in a servlet container such as Jetty or Tomcat. This not only reduces DevOps effort but also the risk of version conflicts, CDN outages, or cross-origin issues.

This server-side approach is particularly suitable for applications whereData sovereignty, security and long-term maintainability are in the foreground – for example, with internal tools, admin consoles, or security-critical services. The integration of session-based security mechanisms, IP filters, or role-based access is also much more controlled in a Vaadin servlet than with headless backends with an external JavaScript frontend.

The choice of Vaadin Flow is therefore not just a technological decision, but an expression of an architectural style:stable, server-centric, Java-enabled – ideal for secure, maintainable and explainable systems.

3. User interface architecture

The user interface is not an isolated component, but an integral part of the system. It must fit seamlessly into the existing module architecture without enforcing domain-specific dependencies or abandoning the loose coupling between core logic and application layers. In a purely Java-based environment without frameworks such as Spring or Jakarta EE, this separation is particularly important because it allows the UI to be viewed as a separate layer, both technically and semantically.

3.1 Integration into the existing WAR structure

The application is already organised as a multi-tiered Maven project, which is consolidated into a central WAR module. This WAR module knows the servlet context, initialises the HTTP server (for REST) and the VaadinServlet (for the UI), and thus takes on the role of Integration and delivery unit. The previous modulesshortener-core(domain) andshortener-api(HTTP routing) remain unaffected.

The new UI module –shortener-ui-required – is implemented as an additional Maven module, which is used exclusively by shortener-core. It contains all Vaadin-specific logic, UI components, and views. In the WAR module, this module is then connected via a servlet registration – typically via a Servlet, which is defined in the deployment descriptor (web.xml) or via ServletContainerInitializer- and is integrated.

The structure follows the principle:Only the WAR module knows the specific deployment topology. All other modules remain generic and independent.

3.2 Separation of routing, views and services

Within the UI module, responsibilities are divided:

• Routing andNavigation are centrally controlled via a MainLayout class and with @Route-annotated views. These define the paths within the UI (e.g./, /create, /admin) and automatically take over session and history management from Vaadin.

• Views (approximately CreateView, OverviewView, StatsView) form the interactive interfaces. They integrate Vaadin components such as TextField, Grid, Button, Dialog and provide user interaction. Each view is stateless and delegates to services.

• Services encapsulate the interaction with the business logic. These are not typical Spring beans, but regular Java classes that can be implemented either directly or via the Singleton pattern (ServiceRegistry, StaticFactory), and are instantiated. They call methods shortener-core and take care of conversion, validation and error handling.

This structure allows for a clear separation between presentation, application logic, and business logic. It is explicitly based on classic UI architectural patterns such as Model-View-Service (MVS), without the overhead or magic of full-fledged frameworks.

3.3 Communication with the core module (DI without frameworks)

Since dependency injection containers are not used, the views are deployed with service instances manually or via a central deployment class. A typical approach is to implement aUiServiceLocator , which contains all the required components (e.g.UrlMappingStore, ShortCodeGenerator), is created once and then made available as a singleton.

For example, the constructor of a view can look like this:

javaCopy

publicclassOverviewViewextendsVerticalLayout{  privatefinalUrlMappingStorestore=UiServiceLocator.getUrlMappingStore();  publicOverviewView(){// Build components, display data, define actions  }}

This form of explicit dependency creation has the advantage that the origin of all components remains traceable, no hidden initialisation takes place, and the entire application context remains transparently controllable – a property that is particularly important in safety-critical systems.

4. User guidance and UX design

An administration interface is not a classic front end for end users, but a specialised tool – comparable to a precision instrument. Its user interface must be efficient, robust, and explainable. In contrast to dynamic consumer UIs, which boast animations, branding, or complex interactions, clarity, predictability, and freedom from redundancy are paramount here. The design of a Vaadin interface for a URL shortener, therefore, follows a minimalist, function-centric UX strategy.

4.1 Navigation concept: Dashboard, Create, Overview

From the user’s perspective, the application is divided into three central task areas:

• Creating new short links : A simple input form with target URL validation and an optional custom alias. Speed and clarity are paramount.
• Overview of existing mappings : A tabular display of all generated shortlinks, supplemented with sorting and filtering options. This allows for targeted searches by alias, target address, or expiration date.
• Administrative Dashboard : An area for viewing metrics (e.g. number of mappings, frequently used short links) as well as for targeted deletion or extension of entries.

Navigation is done via a Vaadin layout with a main menu (e.g.AppLayout orDrawerToggle) and named routes. All routes are directly accessible via URL, with bookmarks and refreshes always leading to a consistent state—an inherent advantage of Vaadin Flow due to server-side session management and declarative routing.

4.2 Input validation and feedback

A key UX criterion is the correctness of the inputs. When creating a new mapping, it must be ensured that

• The input field for the URL is not empty,
• The URL is syntactically valid (including http/https),
• optional aliases are not already assigned or reserved.

These validations are performed immediately upon input (client-side via Binder) and again server-side in the service layer, whereby the view displays corresponding error messages asNotification ,ErrorMessage or directly in the field context. A deliberate UX detail: In the event of errors, the focus remains on the field, the previous input is not deleted, and a visual highlight appears, instead of a general error dialogue.

Successful processes (e.g., successfully created a short link) lead to clear success messages, including immediate copying options, e.g., via a button with an icon (“copy to clipboard”) to support cross-media use.

4.3 Readability, design system and accessibility

The UI is functional, not decorative. Accordingly, attention is paid to a clear visual grid: sufficient white space, consistent labels, understandable icons, and no unnecessary animations. Colours serve solely for semantic marking (e.g., red for deletion actions, green for success messages), not for stylistic individualisation.

Vaadin Flow comes with a solid standard design that extends@Theme, or custom CSS classes can be extended, but only where functionally necessary. The entire application remains fully keyboard-operated, screen reader-compatible, and high-contrast enough for low-barrier use. This is especially important in contexts where administrators work in security-critical areas with increased accessibility requirements.

5. Implementation of the UI components with Vaadin Flow (including source code)

5.1 MainLayout: Navigation frame and routing

The MainLayout forms the common structure of all views. It is based on the AppLayout and defines a vertical navigation menu with links to the central views:

javaCopy

@Theme("shortener-theme")publicclassMainLayoutextendsAppLayout{    publicMainLayout(){        DrawerToggletoggle=newDrawerToggle();        H1title=newH1("URL Shortener Admin");        title.getStyle().set("font-size","var(--lumo-font-size-l)")                         .set("margin","0");        HorizontalLayoutheader=newHorizontalLayout(toggle,title);        header.setDefaultVerticalComponentAlignment(Alignment.CENTER);        header.setWidthFull();        header.setPadding(true);        addToNavbar(header);        RouterLinkcreateLink=newRouterLink("Erstellen",CreateView.class);        RouterLinkoverviewLink=newRouterLink("Übersicht",OverviewView.class);        VerticalLayoutmenu=newVerticalLayout(createLink,overviewLink);        menu.setPadding(false);        menu.setSpacing(false);        menu.setSizeFull();        addToDrawer(menu);    }}

This structure separates navigation logic from application logic. New views only need to be@Route(…, layout = MainLayout.class) and connected.

5.2 CreateView: Input form with validation

The view for creating new short links consists of a TextField for the target URL, an optional alias field, and a button for generating the mapping. Simple validation logic is provided via the binder:

xmlCopy

@Route(value = "", layout = MainLayout.class)public class CreateView extends VerticalLayout {    private final UrlMappingStore store = UiServiceLocator.getUrlMappingStore();    public CreateView() {        setSpacing(true);        setPadding(true);TextField urlField = new TextField("Target URL");        urlField.setWidthFull();        TextField aliasField = new TextField("Alias (optional)");        aliasField.setWidth("300px");Button shortenButton = new Button("Shorten");        shortenButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY);        HorizontalLayout actions = new HorizontalLayout(shortenButton);        actions.setAlignItems(Alignment.END);        Binder<ShortenRequest> binder = new Binder<>(ShortenRequest.class);        ShortenRequest request = new ShortenRequest();        binder.forField(urlField).asRequired("URL must not be empty")              .withValidator(url -> url.startsWith("http://") || url.startsWith("https://"), "Only HTTP(S) URLs allowed")              .bind(ShortenRequest::url, ShortenRequest::url);        binder.forField(aliasField)              .bind(ShortenRequest::alias, ShortenRequest::alias);        shortenButton.addClickListener(click -> {            if (binder.writeBeanIfValid(request)) {                Optional<String> code = createShortCode(request);                code.ifPresentOrElse(c -> {Notification.show("Short link created: " + c);                    urlField.clear();                    aliasField.clear();}, () -> Notification.show("Alias already assigned or error saving", 3000, Position.MIDDLE));            }        });add(new H2("Create new short link"), urlField, aliasField, actions);    }    private Optional<String> createShortCode(ShortenRequest req) {        try {            return Optional.ofNullable(                req.alias() == null || req.alias().isBlank()                ? store.createMapping(req.url()).shortCode()                : store.createCustomMapping(req.alias(), req.url()).shortCode()            );        } catch (IllegalArgumentException e) {            return Optional.empty();        }    }    private record ShortenRequest(String url, String alias) {        public ShortenRequest() { this("", ""); }        public String url() { return url; }        public String alias() { return alias; }    }}

This view encapsulates the entire user interaction for creating new short links. Incorrect entries result in immediate feedback, while valid entries are persisted and output directly as feedback.

5.3 OverviewView: Grid-based management view

The OverviewView displays all existing mappings in the grid. Optionally, additional columns such as creation time or expiration date can be displayed. Each row allows deletion via a confirmation dialogue:

xmlCopy

@Route(value = "overview", layout = MainLayout.class)public class OverviewView extends VerticalLayout {    private final UrlMappingStore store = UiServiceLocator.getUrlMappingStore();    private final Grid<ShortUrlMapping> grid = new Grid<>(ShortUrlMapping.class, false);    public OverviewView() {        setSizeFull();        setPadding(true);        setSpacing(true);        grid.addColumn(ShortUrlMapping::shortCode).setHeader("Short Code");        grid.addColumn(ShortUrlMapping::originalUrl).setHeader("Original URL").setFlexGrow(1);        grid.addColumn(m -> m.createdAt().toString()).setHeader("Erstellt am");        grid.addComponentColumn(this::buildActionButtons).setHeader("Aktionen");        grid.setItems(store.findAll());        grid.setSizeFull();        add(new H2("Alle Kurzlinks"), grid);    }    private Component buildActionButtons(ShortUrlMapping mapping) {Button delete = new Button("Delete", e -> openConfirmDialog(mapping));        delete.addThemeVariants(ButtonVariant.LUMO_ERROR);        return delete;    }    private void openConfirmDialog(ShortUrlMapping mapping) {        Dialog dialog = new Dialog();dialog.add(new Text("Really delete short link? (" + mapping.shortCode() + ")"));        Button confirm = new Button("Ja", e -> {            store.delete(mapping.shortCode());            grid.setItems(store.findAll());            dialog.close();Notification.show("Short link deleted.");        });        confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY);        Button cancel = new Button("Cancel", e -> dialog.close());        dialog.add(new HorizontalLayout(confirm, cancel));        dialog.open();    }}

The grid is updated automatically after deletions. Additional filter or export functions can be easily added based on this structure.

5.4 Consistent behaviour for dialogues and notifications (with code)

In an administration interface for a security-critical service such as a URL shortener, the way the system handles user interactions is crucial. The response to inputs—be they correct, incorrect, or critical—must bepredictable, explainable and context-appropriate. Particularly important are:

• immediate,non-blocking success messages for valid actions,
• precise, field-oriented fault indications in case of incorrect entries,
• intention-confirming dialogues for potentially destructive operations such as deleting mappings.

These three feedback types can be implemented entirely using Vaadin Flow’s built-in tools – specifically: notification, dialogue, and component binding via Binder.

Success stories via Notification

After successful actions (e.g., creating a short link or deleting an entry), a discreet, automatically disappearing notification is displayed. This is non-blocking, clearly informs users of the success, and allows them to continue working seamlessly.

Notification.show(“Short link created successfully.”, 3000, Notification.Position.TOP_CENTER);

For targeted feedback with content (e.g. the generated short code), an inline component or a copyable text block can be displayed alternatively:

kotlinCopy

varshortCodeField=newTextField("Your short link");shortCodeField.setValue("https://short.ly/"+code);shortCodeField.setReadOnly(true);shortCodeField.setWidthFull();add(shortCodeField);Notification.show("You can now copy the link.");

Error messages through field-based validation withBinder

Failed inputs—such as invalid URLs or duplicate aliases—do not need to be displayed in general dialogues, but rather directly in the affected field. To do this, Vaadin uses the Binder mechanism, which binds validation rules directly to UI components:

xmlCopy

Binder<ShortenRequest> binder = new Binder<>(ShortenRequest.class);binder.forField(urlField).asRequired("URL must not be empty").withValidator(this::isValidHttpUrl, "Only valid HTTP(S) URLs allowed")    .bind(ShortenRequest::url, ShortenRequest::url);

The method isValidHttpUrl checks, for example, with a simple Regex orURI.create(…) Logic for syntactical validity. Errors are automatically displayed as error messages directly below the field, visually highlighted and focused:

javaCopy

privatebooleanisValidHttpUrl(Stringurl){    returnurl!=null&&(url.startsWith("http://")||url.startsWith("https://"));}

If an error occurs at the application level (e.g. alias already assigned), this can also be communicated via a global notification:

javaCopy

Notification.show("Alias already taken",3000,Notification.Position.MIDDLE)            .addThemeVariants(NotificationVariant.LUMO_ERROR);

Destructive actions only after explicit confirmation

When deleting a short link, resetting a counter, or performing similar irreversible actions, the operation must never be performed without confirmation. Vaadin’s dialogue component, combined with descriptive text and two explicit buttons, is ideal for this.

Example from the OverviewView:

javaCopy

privatevoidopenConfirmDialog(ShortUrlMappingmapping){    Dialogdialog=newDialog();dialog.setHeaderTitle("Confirm deletion");Spanmessage=newSpan("Do you really want to delete the short link? ("+mapping.shortCode()+")");    dialog.add(message);Buttonconfirm=newButton("Delete",click->{        store.delete(mapping.shortCode());        grid.setItems(store.findAll());        dialog.close();Notification.show("Short link deleted.",3000,Notification.Position.TOP_CENTER)                    .addThemeVariants(NotificationVariant.LUMO_SUCCESS);    });    Buttoncancel=newButton("Abbrechen",event->dialog.close());    cancel.addThemeVariants(ButtonVariant.LUMO_TERTIARY);    dialog.getFooter().add(cancel,confirm);    dialog.open();}

Features of this implementation:

• An explicit button triggers the action.
• Confirmation is dialogue-based with a cancel option.
• The change will be reflected immediately upon completion.
• Feedback is provided immediately via notification.

Interaction and reuse

A central UX pattern of this interface isto standardise the behaviour of all interactions. This avoids surprises for users. Consistent behaviour means:

action	Reaction
URL successfully shortened	Notification, display of the short link
Incorrect input	Field marking, inline error message
Alias already taken	central notification with incorrect colouring
Delete the short link	Dialogue with two buttons
Confirmed deletion	Notification: Grid is reloading

These patterns can be encapsulated component-based – for example, using utility methods for dialogue generation or notification factories, which can later also be adapted for logging or internationalisation.

7. Deploying the UI in the WAR context

The user interface is not delivered as a separate application, but integrated into a central WAR file that contains both the REST API and the Vaadin-based UI. This monolithic approach is deliberately chosen: It enables a consistent deployment model on servlet containers such asJetty ,Tomcat orUndertow without relying on complex build pipelines, containerization, or framework integration.

7.1 Servlet configuration for Vaadin inweb.xml

Vaadin Flow is traditionally registered via the servlet com.vaadin.flow.server.VaadinServlet. Since there is no automatic configuration (as with Spring Boot), the assignment is made explicitly in the deployment descriptor:

xmlCopy

<web-appxmlns="http://xmlns.jcp.org/xml/ns/javaee"         version="4.0">    <!-- I require UI -->    <servlet>        <servlet-name>VaadinServlet</servlet-name>        <servlet-class>com.vaadin.flow.server.VaadinServlet</servlet-class>        <heat-param>            <param-name>ui</param-name>            <param-value>shortener.ui.MainUI</param-value>        </init-param>        <load-on-startup>1</load-on-startup>    </servlet>    <servlet-mapping>        <servlet-name>VaadinServlet</servlet-name>        <url-pattern>/*</url-pattern>    </servlet-mapping></web-app>

The entry**/*** as a URL pattern causes all paths not occupied by the REST API (e.g./shorten, /abc123) to be handled by the UI. The REST endpoints can still be accessed via explicit HttpServlet registrations with priority.

Alternatively, manual division via filters or explicit exclusion lists is possible, for example,/api/* should be strictly reserved for the REST branch.

7.2 Theme-Integration ohne Frontend-Toolchain

A big advantage of Vaadin Flow in server mode is that there isno Node.js, npm or Webpack. All components and resources are handled at the JVM level and delivered as servlet resources at runtime. Themes can be defined directly in the classpath:

javaCopy

@Theme(value="shortener-theme")publicclassMainLayoutextendsAppLayout{...}

In the directory frontend/themes/shortener-theme/, then lies:

• styles.css– own adjustments
• theme.json– Metadata and component binding

The Maven plugin vaadin-maven-plugin automatically generates the corresponding frontend bundle during package or install, fully integrated into the WAR build, without the need for external tools.

A typical Maven excerpt inshortener-war/pom.xml:

xmlCopy

<plugin>    <groupId>com.vaadin</groupId>    <artifactId>vaadin-maven-plugin</artifactId>    <version>${vaadin.version}</version>    <executions>        <execution>            <goals>                <goal>prepare-frontend</goal>                <goal>build-frontend</goal>            </goals>        </execution>    </executions></plugin>

The use of build-frontend. This is only required during initial deployment or when making theme changes. During operation, everything is loaded from the WAR.

7.3 Operation in Tomcat or Jetty

The resulting WAR file can be used in anyServlet 4.0+ compatible container for deployment. Recommended are:

• Jetty 12 for embedded deployments or local tests
• Apache Tomcat 10.1+ for production-related scenarios
• Eclipse Servlet Container (e.g. Open Liberty) for modular applications

Deployment is incredibly simple:

cp target/shortener.war $CATALINA_BASE/webapps/

After starting the container, the application can be accessed via the usual path:

http://localhost:8080/shortener/

REST endpoints (e.g., POST /shorten) and UI routes (/overview, /create) coexist. This results in a robust, clearly controlled deployment model:

• No framework magic
• No container lock-in No build complexity

8. Conclusion and outlook

With the integration of a Vaadin Flow-based interface, our URL shortener not only receives an interactive user interface, but also a complete, maintainable and production-ready management layer – realisedexclusively with funds from the JDK. The graphical UI serves not as a decorative accessory, but as an integral part of the system architecture. It enables the secure creation, analysis, and maintenance of short links – typically via the same infrastructure used for REST endpoints or background processes.

The decision,exclusively Core Java and Vaadin Flow, has proven to be viable: The separation of module boundaries remains clear, deployment takes place as a classic WAR file, and all interactions – from validations and feedback logic to permission models – can be implemented in a comprehensible and transparent manner. Instead of framework magic, convention-over-configuration, or annotation-based autowiring, the focus here is once again onunderstandable system design down to the details.

Particularly noteworthy is the efficiency with which Vaadin Flow maps UI logic directly in the Java context. Developers retain control over state, access protection, and lifecycles – without relying on build toolchains, client routing, or third-party libraries. The associated simplification of the maintenance and security model is a significant advantage in the context of safety-critical infrastructure.

The system can also be further developed in the future without any architectural disruption. Possible next steps include:

• Modularisation of the UI into reading and writing segments
• Asynchronous Event-Analysis via server push or WebSockets
• Connection to external security systems such as OAuth2 or LDAP
• Deployment on embedded Jetty for minimalist distributions
• Migration to native builds via GraalVM to reduce runtime overhead

This makes the URL shortener suitable not only as a learning object or internal tool, but also as a blueprint for lean, robust and self-determined Java web applications – without vendor lock-in, without complex tooling dependencies and with the focus onReadability, transparency and control.

Happy Coding

Sven

]]>JavaVaadinhttps://svenruppert.com/posts/connecting-rest-services-with-vaadin-flow-in-core-java/Tue, 24 Jun 2025 09:39:25 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/connecting-rest-services-with-vaadin-flow-in-core-java/1. Introduction Why REST integration in Vaadin applications should not be an afterthought In modern web applications, communication with external services is no longer a special function, but an integral part of a service-oriented architecture. Even if Vaadin Flow, as a UI framework, relies on server-side Java logic to achieve a high degree of coherence between view and data models, the need to communicate with systems outside the application quickly arises. These can be simple public APIs—for example, for displaying weather data or currency conversions—as well as internal company services, such as license verification, user management, or connecting to a central ERP system.<![CDATA[

1. Introduction

Why REST integration in Vaadin applications should not be an afterthought

In modern web applications, communication with external services is no longer a special function, but an integral part of a service-oriented architecture. Even if Vaadin Flow, as a UI framework, relies on server-side Java logic to achieve a high degree of coherence between view and data models, the need to communicate with systems outside the application quickly arises. These can be simple public APIs—for example, for displaying weather data or currency conversions—as well as internal company services, such as license verification, user management, or connecting to a central ERP system.

The challenge here lies not in technical feasibility. Still, in structural embedding, REST calls should not appear “incidentally” in the view code, but rather be abstracted and controlled in a cleanly encapsulated service layer. Especially in safety-critical or highly available systems, it’s not just access that matters, but its failure behaviour, fallback strategies, and reusability.

1. 1. Introduction
   1. Why REST integration in Vaadin applications should not be an afterthought
2. 2. Architecture overview
   1. Separation of UI, service and integration – best practices without framework ballast
3. 3. HTTP-Client in Java
   1. Introduction to java.net.http.HttpClient as a modern, native solution
4. 4a. Der REST-Service als Adapter
   1. Building a lean Java class for REST communication with object mapping
5. 4b. Der REST-Endpoint in Core Java
   1. How to create an HTTP endpoint without frameworks with minimal effort
   2. Minimal example: REST endpoint that returns JSON
   3. What is happening here, and why does it make sense
   4. Outlook: Modular expansion of REST servers
6. 5. Type-safe data models with records
   1. How Java Records provide clarity and security
7. 6. Error handling and robustness
   1. Handling status codes, IO problems and termination conditions
   2. HTTP status codes as a starting point
   3. Dealing with IO errors
   4. Logging with Thought
   5. Retry and Timeout
   6. Control instead of surprise
8. 7. Integration in die Vaadin-UI
   1. Example use in the view layer – synchronous and understandable
   2. UI response to errors
      1. Visible feedback without blocking interaction
   3. Asynchronous extension (optional)
9. 8. Production-ready safeguards
   1. Authentication, time limits, retry logic and logging – what you should pay attention to
   2. 8.1 Authentication: Headers instead of framework magic
   3. 8.2 Time limits: Protection against hanging services
   4. 8.3 Retry strategies: controlled and limited
   5. 8.4 Logging and Correlation
   6. 8.5 Resilience through convention
10. 9. Asynchronous extension with CompletableFuture 1.When and how to integrate non-blocking HTTP calls into UI workflows 2.Starting point: Blocking REST calls 3.Solution: Non-blocking via CompletableFuture 4.Error handling in the futures chain 5.Progress indicator and UI states 6.Combination with multiple requests
11. 10. Conclusion and outlook 1.REST adapters as a stable bridge in service-oriented architectures 2.Outlook: REST in modular and service-oriented Vaadin applications 3.What remains?

Vaadin Flow offers no built-in mechanism for this, and that’s a good thing. By deliberately avoiding magical abstractions or framework coupling (such as Spring RestTemplate or Feign Clients), it allows maximum control over REST integration. With Java 24 and the now maturejava.net.http.HttpClient, all required building blocks are available directly in the Core JDK – performant, maintainable and framework-independent.

This chapter, therefore, marks the beginning of a series of articles that demonstrate how to connect a Vaadin Flow application to an external REST endpoint, without external dependencies, but with a clear focus on readability, architecture, and security. The goal is not only to write functioning HTTP calls, but also to understand REST as part of a clean, modular software architecture.

2. Architecture overview

Separation of UI, service and integration – best practices without framework ballast

A clear architecture is the foundation of any maintainable application, regardless of whether it is monolithic, modular, or service-oriented. This is especially true for Vaadin Flow, as UI components are defined server-side in Java, thus eliminating any rigid boundaries between the underlying layers. This proximity between the user interface and backend logic is one of Vaadin’s most significant advantages, but it also presents an architectural challenge when integrating external systems.

The integration of a REST endpoint should therefore always be done via an intermediary service layer. This refers to a dedicated “adapter” that, on the one hand, encapsulates the connection to the external REST service and, on the other hand, provides a type-safe, stable API for the UI layer. The advantage is obvious: Changes to the format of the remote system or the protocol behaviour do not affect the application structure or the user interface—they remain isolated in the adapter module.

This separation can be elegantly implemented in a Vaadin project without additional frameworks. A standard layered structure consists of the following components:

• UI layer: It contains the Vaadin views and components that work exclusively with Java data objects. No network communication or JSON processing takes place here.
• Service shift: It mediates between the UI and integration, potentially coordinates multiple adapters, and is responsible for domain-specific business logic. This layer is unaware of the technical details of the REST protocol.
• Adapter layer: It provides the actual REST access. This includes HTTP calls, header management, timeout handling, JSON deserialization, and error evaluation. This layer has no UI classes or interactive logic.

This modular, three-part structure enables a testable and extensible framework. It enables the targeted simulation of REST access in unit tests or the use of alternative implementations (e.g., for offline modes or mocking in the development system) – all without requiring refactoring of the UI code.

Especially in Java 24, this principle can be excellently complemented with records, sealed types, and functional interfaces, making the resulting architecture not only stable but also concise and modern. The following figure (not included) illustrates how the data flows between the layers, from the UI request to the service, to the adapter, and back – always clearly separated but tightly interlinked by typed interfaces.

With this foundation in mind, we will discuss the technical implementation of REST communication in the next chapter, specifically using theHttpClient , which has been part of the JDK since Java 11 and has now established itself as a high-performance, well-tested standard.

3. HTTP-Client in Java

Introduction tojava.net.http.HttpClient as a modern, native solution

Since Java 11, the java.net.http.HttpClient, a powerful, standards-compliant, and thread-safe HTTP client, is available and fully included in the JDK. In Java, this client has long been established as a reliable means of REST communication, especially in projects that deliberately avoid frameworks to achieve maximum control, portability, and predictability.

In contrast to previous solutions, such asHttpURLConnection , which were laborious and error-prone, the modernHttpClient features a straightforward, fluent-based API that supports both synchronous and asynchronous communication usingCompletableFuture. It is resource-efficient, supports automatic redirects, HTTP/2, and can be equipped with configurable timeouts, proxies, and authentication.

In practice, the use of the HttpClient begins with its configuration as a reusable instance:

javaCopy

HttpClientclient=HttpClient.newBuilder()    .connectTimeout(Duration.ofSeconds(5))    .version(HttpClient.Version.HTTP_2)    .build();

The HttpClient is immutable and thread-safe. Once created, it can be used for any number of requests. This is especially crucial in server applications, as repeated regeneration would not only be inefficient but also potentially problematic for resource utilisation.

For the specific request, aHttpRequest that encapsulates all parameters such as URI, HTTP method, headers and optional body data:

sqlCopy

HttpRequestrequest=HttpRequest.newBuilder()    .uri(URI.create("https://api.example.com/data"))    .header("Accept","application/json")    .GET()    .build();

The execution is synchronous with:

HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString());

Alternatively, the methodsendAsync() can be used to realise non-blocking interactions based on CompletableFuture – a model that can be particularly useful for UI-oriented or reactive applications when blocking HTTP processing should be avoided.

What makes this API particularly special is the strict separation of request construction, client configuration, and result handling—no hidden magic, reflection, or XML-based configuration is used. Control lies entirely with the developer, and the API is understandable and type-safe.

The return is always made via aHttpResponse, where the type T is determined by the selected BodyHandler—typically String, byte[], InputStream, or Void. This separation allows for efficient processing of both simple text responses and binary data.

In the next chapter, I’ll show how this HTTP infrastructure can be converted into a production-ready adapter class, including header handling, error checking, and JSON deserialization into Java records. The goal is not just functional communication, but maintainable, robust, and testable code.

4a. Der REST-Service als Adapter

Building a lean Java class for REST communication with object mapping

After the technical basics of theHttpClient are established in Java 24, the central question arises: How can REST calls be encapsulated so that they remain reusable, extensible, and UI-independent? The answer lies in the adapter principle, explicitly implemented as a standalone service class that encapsulates access to a specific REST endpoint and provides a type-safe API for the application core.

Such an adapter is responsible for:

• Structure and execution of the HTTP request,
• Interpretation of the HTTP response code,
• Transformation of the response data (e.g. JSON) into a Java model,
• optional: logging, header management, error handling and retry logic.

However, these tasks shouldn’t be performed in the UI layer or presenter classes. Instead, a final-declared Java class is recommended, whose methods are specifically designed to load or send specific domain-specific data objects – for example:fetchUserData() ,submitOrder() ,validateLicenseKey().

A minimal adapter for a GET call with a JSON response could look like this:

xmlCopy

import java.io.IOException;import java.net.URI;import java.net.http.*;import java.time.Duration;import com.fasterxml.jackson.databind.ObjectMapper;import com.example.model.DataObject;public final class ExternalApiService {    private final HttpClient client;    private final ObjectMapper mapper;    private final URI endpoint;    public ExternalApiService(String baseUrl) {        this.client = HttpClient.newBuilder()                .connectTimeout(Duration.ofSeconds(5))                .build();        this.mapper = new ObjectMapper();        this.endpoint = URI.create(baseUrl + "/data");    }    public DataObject fetchData() throws IOException, InterruptedException {        HttpRequest request = HttpRequest.newBuilder()                .uri(endpoint)                .header("Accept", "application/json")                .GET()                .build();        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());        if (response.statusCode() != 200) {throw new IOException("Unexpected status code: " + response.statusCode());        }        return mapper.readValue(response.body(), DataObject.class);    }}

The aim of this class isnot to be flexible for any endpoint or generic API, but rather to beconcrete and targeted to serve exactly one external use case. This clarity is an advantage, not a disadvantage: It facilitates refactoring, testability, and secure extension if, for example, new headers, authentication mechanisms, or error codes are introduced.

The class encapsulates the entire technical aspect of HTTP communication. It is entirely independent of the UI and has no views, components, or user interactions. It can therefore be easily integrated into JUnit tests—either directly or via interface derivation for mocking.

The record class usedDataObject serves as a transfer object that is automatically deserialised from JSON by the ObjectMapper:

public record DataObject(String id, String value) {}

This combination of a concise object structure and a stable communication layer forms the backbone of a clean REST integration—without frameworks, without magic, but with a clear separation of responsibilities. Errors can be specifically intercepted, additional parameters can be easily incorporated, and the architecture remains transparent.

4b. Der REST-Endpoint in Core Java

How to create an HTTP endpoint without frameworks with minimal effort

A REST endpoint is essentially nothing more than a specialised HTTP handler. While Spring or JakartaEE use controllers, annotations, and automatic serialisation, a pure Java approach requires a bit more manual work, but is rewarded with complete control over behaviour, performance, security boundaries, and dependencies.

Since Java 18, thecom.sun.net.httpserver.HttpServer API, a lightweight HTTP server, has been available. It is ideal for simple REST endpoints—for example, as a test mock, an internal microservice, or in this case, as a local data source for Vaadin. In combination with a JSON mapping (e.g.ObjectMapper from Jackson), it creates a REST backend without any external platform.

Minimal example: REST endpoint that returns JSON

The following class starts an HTTP server on port 8080, which, in a GET on/data, returns a JSON object:

javaCopy

importcom.sun.net.httpserver.HttpServer;importcom.sun.net.httpserver.HttpHandler;importcom.sun.net.httpserver.HttpExchange;importcom.fasterxml.jackson.databind.ObjectMapper;importjava.io.IOException;importjava.io.OutputStream;importjava.net.InetSocketAddress;publicclassSimpleRestServer{    publicstaticvoidmain(String[]args)throwsIOException{        HttpServerserver=HttpServer.create(newInetSocketAddress(8080),0);        server.createContext("/data",newDataHandler());        server.setExecutor(null);// default executor        server.start();System.out.println("Server is running on http://localhost:8080/data");    }    staticclassDataHandlerimplementsHttpHandler{        privatefinalObjectMappermapper=newObjectMapper();        @Override        publicvoidhandle(HttpExchangeexchange)throwsIOException{            if(!"GET".equals(exchange.getRequestMethod())){                exchange.sendResponseHeaders(405,-1);// Method Not Allowed                return;            }            vardata=newDataObject("abc123","42.0");            Stringresponse=mapper.writeValueAsString(data);            exchange.getResponseHeaders().add("Content-Type","application/json");            exchange.sendResponseHeaders(200,response.getBytes().length);            try(OutputStreamos=exchange.getResponseBody()){                os.write(response.getBytes());            }        }    }    publicrecordDataObject(Stringid,Stringvalue){}}

What is happening here, and why does it make sense

• The server listens on port 8080 and accepts requests/data in contrast to.
• Only GET requests are accepted; all other HTTP methods receive a status code of 405.
• The result object is an ObjectMapper-serialised instance – this is the same data type (DataObject) that is also used in the REST adapter on the client side.
• The answer will be delivered with the appropriate Content-Type, specifically application/json.

This implementation is minimal but functional. It can be tested immediately, run locally, and queried from the Vaadin UI via the adapter described in Chapter 4a. It requires no XML, no DI container, no JAR hierarchies—just plain Java.

Outlook: Modular expansion of REST servers

A REST server based on this principle can be easily expanded:

• More createContext(…)-Handler for additional endpoints
• Support for POST, PUT, DELETE with Payload-Parsing
• Path parameters through simple URI parsing
• Authentication logic via header evaluation
• Logging, metrics and request correlation as an extension
• Outsourcing to modules if multiple services are to be created

This setup can be used particularly elegantly in development and testing, for example, to simulate external services or to reproduce specific error cases (404, 500).

5. Type-safe data models with records

How Java Records provide clarity and security

The data model plays a central role in the communication between a Vaadin Flow application and an external REST service. It bridges the gap between the JSON representation of the remote resource and the object representation processed in Java. The clearer, more type-safe, and immutable this model is, the more robust the application architecture becomes, especially when the REST interface changes or when used in parallel UI contexts.

Since Java 16, so-calledRecords – a language feature explicitly designed for this type of data structure: compact, immutable objects with semantically unambiguous data transport characteristics. A record is not a replacement for a complete domain entity with behaviour, but the ideal representation forstructured response data from REST endpoints.

An example: A REST service delivers JSON responses like

jsonCopy

{  "id":"abc123",  "value":"42.0"}

A suitable Java record to represent this structure looks like this:

public record DataObject(String id, String value) {}

This record is:

• Unchangeable : Its fields are final and publicly readable, but not modifiable.
• Comparable : equals() and hashCode() are automatically implemented correctly.
• Structured : The signature also serves as documentation of the expected JSON format.
• Compact : Compared to classic POJOs, there is no boilerplate code.

Deserialization from JSON with anObjectMapper(as shown in Chapter 4) works directly and without further annotations as long as the field names match. This promotes readability and reduces the risk of hidden deserialization errors.

Additionally, records are well-suited for use asDTOs (Data Transfer Objects). For example, when multiple external REST endpoints deliver different aspects of the same domain concept, each with its own response structure. Strict type safety of records then protects against unintentional mixing or incorrect field usage.

Records can also be nested to represent more complex JSON structures—for example, when an object contains a list of other objects or provides structured metadata. Here, too, developers benefit from the expressiveness, readability, and stability that records offer over traditional getter and setter classes.

In the context of a Vaadin application, this has an immediate positive effect: Records can be used directly in UI components, for example, for display inGrid ,FormLayout or custom components. The data objects themselves contain no presentation logic, but remain purely structural—and this is precisely what is desirable in a separated architecture.

In the next chapter, we will demonstrate how this structured REST communication can be utilised in the UI layer and identify patterns that have proven successful in practice for effectively linking error handling, response logic, and user interaction.

6. Error handling and robustness

Handling status codes, IO problems and termination conditions

In practice, a REST call is more than just an HTTP request. It is an insecure operation over a potentially unstable medium, with numerous possible sources of error, including network problems, timeout behaviour, invalid response formats, rejected requests, or temporary server errors. The quality of a REST integration is therefore evident,not in successful communication , but instead in its behaviour in the event of a mistake.

Especially in Vaadin Flow applications that remain active on the server side for extended periods, a single failed REST call can result in a view not being initialised correctly, a user action being suspended, or incomplete data being displayed. Therefore, the goal is to implement error handling that:

• semantically distinguishes between transport-related and technical errors,
• provides defined return values ​​or exceptions to which the UI can react specifically,
• and established a comprehensible logging strategy.

HTTP status codes as a starting point

Even the interpretation of the response code requires care. While200 OK and201 Created usually indicate success, other codes such as204 No Content ,404 Not Found, or409 Conflict can be entirely intentional – and should not be treated as errors across the board. The business context is crucial: A “not found” error can be deliberate and trigger a specific UI response, such as an empty display or an alternative form.

The adapter should therefore not perform generic error handling across all code, but rather explicitly evaluate what makes sense in the respective application scenario. Example:

javaCopy

if(response.statusCode()==404){  returnOptional.empty();// targeted reaction to missing resource}elseif(response.statusCode()!=200){  thrownewIOException("Unexpected status: "+response.statusCode());}

Dealing with IO errors

Transport errors – such as timeouts, DNS problems, or connection failures – typically triggerIOException orInterruptedException. These should not be swallowed in the adapter orRuntimeException. Instead, the caller is signalled that the response is not usable. An error message can then be displayed in the UI without the application entering an undefined state.

Logging with Thought

Errors that cannot be explained by user interaction should be logged on the server side, but not necessarily displayed to the user. A shortLogger.warn(…) An entry can help here, for example, to make external API errors traceable without flooding log files with irrelevant details. It is recommended to use different log levels depending on the error type: INFO for normal non-detectable behaviour, WARN for inconsistent responses, and ERROR only for real failures or bugs.

Retry and Timeout

In production systems, it can be useful to automatically retry certain errors (e.g., 503 Service Unavailable or IOException) with backoff and limitation. However, this logic should not be implemented in the UI code, but rather in the adapter or a delegated “RetryHandler.” Timeouts should also be set explicitly:

javaCopy

HttpRequest.newBuilder()    .timeout(Duration.ofSeconds(3))    .build();

Without defined time limits, a REST call can inadvertently lead to a blocking UI thread, especially when called synchronously.

Control instead of surprise

The central goal of any error handling is to ensure thePredictability ofbehaviour : A REST adapter should always provide defined semantics—either a correct result object, a declared exception, or an optional error that is explicitly checked. No hidden null values, no logic in catch blocks, no silent terminations.

The resulting UI remains able to react in a controlled manner, whether by displaying notifications, activating a retry button, or switching to offline data. The result is not only a more robust architecture but also a better user experience.

In the next chapter, I will show how these REST results are integrated into the UI layer of a Vaadin application, including concrete examples of loading operations, state changes, and error messages in the user context.

7. Integration in die Vaadin-UI

Example use in the view layer – synchronous and understandable

The clean separation between REST communication and UI logic is a central principle of maintainable software architecture. However, there must come a point where external data actually ends up in the user interface—be it in a table, a form, or as part of an interactive visualisation. The integration of the REST adapter into the Vaadin UI is ideally done in this case.synchronous, type-confident and consciously visible in the code – instead of hidden “magic” or automatic binding.

At the centre is a view, e.g., a class that isa VerticalLayout orComposite

and inherits. The REST adapter can be called within this view, typically when constructing the view or during a targeted user interaction. The following example demonstrates a simple use case: When loading the view, a dataset should be retrieved from an external REST service and displayed.

javaCopy

publicclassDataViewextendsVerticalLayout{    privatefinalExternalApiServiceapiService;    publicDataView(){        this.apiService=newExternalApiService("https://api.example.com");        try{            DataObjectdata=apiService.fetchData();            showData(data);        }catch(IOException|InterruptedExceptione){            showError(e.getMessage());        }    }    privatevoidshowData(DataObjectdata){        add(newSpan("ID: "+data.id()));        add(newSpan("Wert: "+data.value()));    }    privatevoidshowError(Stringmessage){Notification.show("Error loading data: "+message,5000,Notification.Position.MIDDLE);    }}

This type of integration is deliberately kept simple, but it demonstrates a central principle: The REST adapter is anexplicit component of view initialisation. There’s no hidden binding, no automatic magic, just traceable, step-by-step data flow. This not only facilitates debugging but also makes state transitions visible and controllable.

UI response to errors

Visible feedback without blocking interaction

The integration of an external REST endpoint into a Vaadin Flow application must not only be technically reliable, but it must also, and most importantly, behave predictably and stably within the user interface. In production scenarios, it is perfectly normal for REST services to be temporarily unavailable, provide slow responses, or fail with an error status, such as500 Internal Server Error, 403 Forbidden, or429 Too Many Requests. What matters is not the error itself, but how it affects the user experience.

In the example shown, a non-blocking notification is used in the event of an error. It appears in the centre of the screen and briefly informs the user that the data loading failed. This approach is recommended for several reasons:

1. The user remains able to act :
   A notification doesn’t interrupt the interaction flow. No dialogue opens, no loading process freezes, and no navigation is blocked. The application remains fully usable, which is especially beneficial in multi-part views or for later, optional loading operations.

2. **The error message is visible in context: **Instead of hiding technical details in logs or simply not displaying “something,” the user is actively informed about the error—transparently, but without technical depth. The information isn’t intended for analysis, but rather to correct expectations:“Something should have appeared here, but is currently unavailable.” _
   _

3. **The application does not exit into an undefined state: **There’s no exception chaining into the UI thread, no empty interface without explanation, and no sudden switching to another route. The user experience remains consistent, ​​even in the event of an error.

For more sophisticated scenarios, further reaction patterns are also available:

• **Retry button or “Try again” action: **A button can be displayed directly below the error message to re-execute the REST call. This leaves repeatability up to the user—useful, for example, in cases of temporary network errors or rate-limited APIs.

• **Fallback display or partial information: **If previous data (e.g., from local storage or a previous session) is available, it can be displayed as a placeholder, along with a note that the current data cannot be loaded at the moment.

• **Visual placeholders or skeleton layouts: **Instead of just displaying empty components, a visual framework can be displayed – for example, grey bars that indicate the expected content but symbolise it:“These data are not yet available.” _
  _

• **Consistent error design across the entire application: **It’s recommended to establish a central component or utility class for displaying errors that’s used system-wide. This ensures consistent display regardless of whether a REST error occurs in the dashboard, a dialogue, or a detailed view.

In summary, this approach pursues one central goal:Errors may occur, but they should not take over the control flow. The UI remains in the hands of the user, not the infrastructure. This creates a resilient interface that looks professional and inspires trust, even when systems in the background aren’t working perfectly.

Asynchronous extension (optional)

In some cases, asynchronous integration can be useful – for example, when dealing with long loading times or interactions that must not block. This can also be achieved with Vaadin and theHttpClient. A typical strategy is to load the data in a background thread and then useUI.access(…) back to the UI thread:

javaCopy

UIui=UI.getCurrent();CompletableFuture.supplyAsync(()->{    try{        returnapiService.fetchData();    }catch(Exceptione){        thrownewCompletionException(e);    }}).thenAccept(data->ui.access(()->showData(data)))  .exceptionally(ex->{      ui.access(()->showError(ex.getCause().getMessage()));      returnnull;  });

This variant is not necessary, but it is helpful in scenarios with many concurrent REST requests or slow-responding APIs.

Integrating a REST adapter into a Vaadin flow view is best done in a structured, explicit, and UI-specific manner. The view is not responsible for constructing the request or interpreting the HTTP code—it only consumes the Java objects provided by the adapter. Errors are handled, states are modelled consciously, and the user interface remains responsive and understandable.

In the next chapter, we’ll take a look at production-ready extensions: authentication, header handling, retry strategies, and logging—everything that makes REST access robust and secure.

8. Production-ready safeguards

Authentication, time limits, retry logic and logging – what you should pay attention to

A REST adapter that works reliably in development and test environments is far from production-ready. As soon as a Vaadin Flow application is embedded in real infrastructures—with access to external APIs, network latencies, security policies, and operational requirements—additional protection mechanisms must be established. These not only address error cases, but also, and most importantly,Non-fault cases underchallenging conditions, such as authentication, latency, rate limiting, logging requirements, or the protection of sensitive data.

The following discussion demonstrates how production-ready REST adapters can be developed using the resources of the Java Development Kit (JDK).

8.1 Authentication: Headers instead of framework magic

External APIs typically require authentication, which can be in the form of a static token, a Basic Auth combination, or an OAuth 2.0 bearer token. In production-based scenarios, multiple variants are often used simultaneously – for example, an API that expects separate public and admin keys.

Instead of shifting this logic to the HTTP client construction, it is recommended to provide your helper methods in the adapter, for example:

javaCopy

privateHttpRequest.BuilderauthenticatedRequest(URIuri){    returnHttpRequest.newBuilder()            .you(s)            .header("Authorization","Bearer "+tokenProvider.getCurrentToken());}

This not only isolates the technical mechanism (header), but also abstracts the source of supply (e.g., token rotation, expiration time). The token itself can be renewed regularly, read from a secrets store, or dynamically calculated, without affecting the call point in the UI code.

8.2 Time limits: Protection against hanging services

Without explicitly set time limits, theHttpClient theoretically has unlimited waiting time for a response, which can be fatal in server-side applications. To control latency and protect UI threads, timeouts should be set both in the client and per request:

javaCopy

HttpClientclient=HttpClient.newBuilder()        .connectTimeout(Duration.ofSeconds(5))        .build();HttpRequestrequest=HttpRequest.newBuilder()        .timeout(Duration.ofSeconds(3))        .type(...)        .GET()        .build();

This separation allows global parameters to be differentiated from the behaviour of individual calls – e.g., for particularly sensitive endpoints or third-party interfaces with historically fluctuating response times.

8.3 Retry strategies: controlled and limited

Not all errors mean that a request fails permanently. Temporary DNS problems can often be intercepted by targeted retries**,** which can resolve503 Service Unavailable errors or connectionless gateways – provided the retries are limited, staggered in time, and context-dependent.

An example of asimple Retry loop without an external library:

javaCopy

publicDataObjectfetchDataWithRetry()throwsIOException{    intattempts=3;    for(inti=1;i<=attempts;i++){        try{returnfetchData();// normal method with HttpClient        }catch(IOException|InterruptedExceptione){if(i==attempts)thrownewIOException("Maximum attempts reached",e);            try{Thread.sleep(i*500L);// linear backoff            }catch(InterruptedExceptionie){                Thread.currentThread().interrupt();thrownewIOException("Retry aborted",ie);            }        }    }    thrownewIllegalStateException("Unreachable code");}

Retry logic must never grow uncontrollably, so that it performs logging and that it is only active in the case of explicitly temporary errors, not in the case of401 ,403 or404.

8.4 Logging and Correlation

In production-ready systems, REST calls are a relevant component of auditing, error analysis, and performance monitoring. Therefore, each call should be systematically recorded in the log, but in a differentiated manner:

• DEBUG : technical details such as URIs, headers, and response sizes
• INFO : regular REST call with semantic meaning (e.g. license check, status change)
• WARN : temporary errors or unexpected responses
• ERROR : permanent errors, incomplete response data, uncaught exceptions

Additionally, a correlation token should be included with each request, for example, as a UUID in theX-Correlation-ID-Header , allowing REST calls to be traced across multiple systems. This can also be encapsulated centrally in the adapter:

private HttpRequest.Builder withCorrelation(HttpRequest.Builder builder) {

** return builder.header(“X-Correlation-ID”, UUID.randomUUID().toString());**

}

8.5 Resilience through convention

A stable REST adapter is not “intelligent” in the sense of being dynamic, butsomewhat predictable, complete, and conservative. Every response is either a result or an exception—never a silence. The data formats are stable and defensively parsed. If expectations are not met, a defined termination occurs. This protects both the UI logic and the user experience, forming the basis for maintainable systems with precise error semantics.

9. Asynchronous extension with CompletableFuture

When and how to integrate non-blocking HTTP calls into UI workflows

In Vaadin Flow applications, the code is traditionally structured synchronously – views respond to user actions, load data, and display results. However, as soon as REST calls come into play, which can take longer than a few milliseconds, the question inevitably arises:How can I offload HTTP communication without blocking the UI thread – and without having to resort to a reactive framework?

The answer to this is provided by the JDK integrated classCompletableFuture. It allows you to declaratively model asynchronous workflows, precisely control concurrency, and return the result to the UI in a controlled manner upon success or failure. In conjunction with Vaadin’s server-side UI model, only one measure is crucial: access to UI components must occur in the correct thread context.

Starting point: Blocking REST calls

In a synchronous example, the call typically looks like this:

javaCopy

try{   DataObjectdata=apiService.fetchData();// synchronous call   showData(data);// direct display in the UI}catch(IOException|InterruptedExceptione){    showError(e.getMessage());}

As long as response times are low and the view is already being constructed synchronously, this is entirely unproblematic. It becomes critical when REST calls follow user actions—for example, after clicking “Refresh” or during filter operations on large data sets. Here, a blocking call would lead to a noticeable delay in the UI.

Solution: Non-blocking via CompletableFuture

To offload the REST call without blocking the UI, the access can be moved to a separate background thread, and the result can be safely returned to the UI later:

javaCopy

UIui=UI.getCurrent();CompletableFuture.supplyAsync(()->{    try{        returnapiService.fetchData();    }catch(IOException|InterruptedExceptione){        thrownewCompletionException(e);    }}).thenAccept(data->ui.access(()->showData(data)))  .exceptionally(ex->{      ui.access(()->showError(ex.getCause().getMessage()));      returnnull;  });

What is happening here is architecturally remarkable:

• supplyAsync(…) starts the call in a separate thread of the Common ForkJoinPool.
• ThefetchData() runs independently of the UI thread – does not block it.
• The result (or error) is then sent viaUI.access(…) and brought back into the server-side context where Vaadin UI components can be safely manipulated.

This separation of calculation (REST call) and presentation (Vaadin components) corresponds to a classic principle of responsive design, only on the server-side level.

Error handling in the futures chain

Another advantage: Error handling is modelled clearly and separately, via**. exceptionally(…)** The original exception object is available within this method. In many cases, a targeted display for the user is sufficient; ​​however, logging, metric collection, or retry logic can be added if necessary.

Progress indicator and UI states

In asynchronous scenarios, it’s recommended to also provide visual feedback about the loading status—for example, through a spinner, a progress bar, or the targeted deactivation of interaction elements. Example:

javaCopy

Buttonrefresh=newButton("Neu laden");refresh.setEnabled(false);add(newProgressBar());CompletableFuture.supplyAsync(()->apiService.fetchData())    .thenAccept(data->ui.access(()->{        showData(data);        refresh.setEnabled(true);    }))    .exceptionally(ex->{        ui.access(()->{showError("Loading failed");            refresh.setEnabled(true);        });        returnnull;    });

Combination with multiple requests

Parallel REST calls can also be made byCompletableFuture.allOf(…) Orchestrate. Data from multiple services can be loaded independently and then analysed together. This technique is beneficial for dashboards, complex forms, or multi-API compositions.

Conclusion of this chapter:
CompletableFuture provides an elegant way to integrate REST calls into Vaadin in a non-blocking manner, without any additional libraries. The control is type-safe and has minimal overhead. Combined with Vaadin’sUI.access(…) mechanism, this creates reactive yet deterministic behaviour that is both technically and ergonomically compelling.

10. Conclusion and outlook

REST adapters as a stable bridge in service-oriented architectures

The integration of external REST services into Vaadin Flow applications is far more than a technical detail. It is a conceptual component of a modular, maintainable, and externally communicative application. This blog post has shown how a REST connectionwithout framework dependencies can be implemented in a structured and robust manner, purely using the onboard resources of the Java Core JDK from version 11, in idiomatic form for Java 24.

The focus is on an adapter that has three key features:

1. Technical clarity: The adapter takes full responsibility for handling HTTP requests, deserialising JSON, error handling, and optional authentication. The UI layer remains completely decoupled from it.

2. Type safety and predictability: The response data is cast into records – compact, immutable data structures that ensure readability, consistency and clean debugging.

3. Error robustness and extensibility: With time limits, logging, retry strategies and asynchronous access, the adapter can be made production-ready – without magic, but with conscious design.

What initially starts as a simple way to “just add an HTTP call” to an application becomesarchitecturally viable communication module This approach is particularly advantageous for applications that grow over time: The REST adapter can evolve into a module that bundles various external systems, maintains version control, remains testable, and still interacts cleanly with the UI.

Outlook: REST in modular and service-oriented Vaadin applications

With increasing complexity and system size, the need to distribute responsibilities grows: Data storage, business logic, UI, and integrations should be independently deployable, testable, and versionable. A typical architecture then moves toward:

• Backend-first architecture with Vaadin as UI facade,
• Microservice tailoring , where each external service is modelled as a port/adapter combination,
• Domain-centric UI , where views explicitly receive data via REST-controlled use cases,
• Security-by-Design , where REST communication is secured via tokens, headers and protocol standardisation.

In all these scenarios, the REST adapter remains a central link – lean, deliberately modelled, but with precise semantics. Precisely because Vaadin Flow doesn’t attempt to automatically abstract or generically bind REST, the developer retains maximum control over requests, data structures, lifecycles, and user interaction.

What remains?

Anyone building a Vaadin application that consumes REST should not treat REST as a workaround or side path, but as anequal interface to the world outside the UI. A stable, testable REST connection is a quality feature. And it can be achieved in Core Java with surprisingly few resources – if it is implemented consciously, structured, and in transparent layers.

Happy Coding

]]>JavaUncategorizedVaadinhttps://svenruppert.com/posts/part-ii-urlshortener-first-implementation/Fri, 20 Jun 2025 12:36:28 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/part-ii-urlshortener-first-implementation/1. Introduction to implementation 1.1 Objectives and differentiation from the architectural part The first part of this series focused on theory: We explained why a URL shortener is not just a convenience tool, but a security-relevant element of digital infrastructure. We discussed models for collision detection, entropy distribution, and forwarding logic, as well as analysed architectural variants – from stateless redirect services to domain-specific validation mechanisms.<![CDATA[

1. Introduction to implementation

1.1 Objectives and differentiation from the architectural part

The first part of this series focused on theory: We explained why a URL shortener is not just a convenience tool, but a security-relevant element of digital infrastructure. We discussed models for collision detection, entropy distribution, and forwarding logic, as well as analysed architectural variants – from stateless redirect services to domain-specific validation mechanisms.

1. 1. Introduction to implementation
   1. 1.1 Objectives and differentiation from the architectural part
   2. 1.2 Technological guardrails: WAR, Vaadin, Core JDK
   3. 1.3 Overview of the components
2. 2. Project structure and module organisation
   1. 2.1 Structure of a modular WAR project
   2. 2.2 Separation of domain, API and UI code
   3. 2.3 Tooling und Build (Maven, JDK 24, WAR-Plugin)
3. 3. URL encoding: Base62 and ID generation
   1. 3.1 Design of a stable short link scheme
   2. 3.2 Implementation of a Base62 encoder
   3. 3.3 Alternatives: Random, Hashing, Custom Aliases
   4. 3.4 Implementation: Base62Encoder.java
   5. 3.5 What is happening here – and why?
4. 4. Mapping Store: Storage of the mapping
   1. 4.1 Interface-Design: UrlMappingStore
   2. 4.2 In-memory implementation with ConcurrentHashMap
   3. 4.3 Extensibility for later persistence
   4. 4.4 Implementation
   5. 4.5 Why so?
5. 5. HTTP API with Java tools
   1. 5.1 HTTP-Server mit com.sun.net.httpserver.HttpServer
   2. 5.2 POST /shorten: Shorten URL
   3. 5.3 GET /{code}: Redirect to the original URL
   4. 5.4 Implementation
      1. Starting point: ShortenerServer.java
      2. POST Handler: ShortenHandler.java
      3. GET-Handler: RedirectHandler.java
      4. JsonUtils.java (Minimal Java JSON without external libraries)
   5. 5.6 Core Java Client Implementation
   6. 5.7 Summary

This second part now turns to the concrete implementation. We develop a first working version of a URL shortener inJava 24 , consciouslywithout the use of frameworks such as Spring Boot or Jakarta EE. The goal is to achieve a transparent, modularly structured solution that provides all core functions: URL shortening, secure storage of mappings, HTTP forwarding, and optional presentation via a Vaadin-based user interface.

Particular attention is paid to the clean separation between encoding, storage, API, and UI. The entire application is delivered as a monolithic artefact – specifically, aclassic WAR file , which is compatible with standard servlet containers such as Jetty or Tomcat. This decision enables rapid deployment and facilitates onboarding and testability.

1.2 Technological guardrails: WAR, Vaadin, Core JDK

The implementation is based on a modern, yet deliberately lean technology stack. Only the built-in tools of the JDK and Vaadin Flow are used as the UI framework. The decision to use Vaadin is based on the requirement to implement interactive administration interfaces without additional JavaScript or separate front-end logic, entirely in Java.

The project isa multi-module structure. The separation between core logic, API layer, and UI remains visible and maintainable in the code. Maven is used as the build tool, supplemented by a WAR packaging plugin that creates a classic servlet deployment structure. The use of Java 24 enables the utilisation of modern language tools, including records, pattern matching, sequenced collections, and virtual threads.

The goal is a production-oriented, comprehensible implementation that can be used both as a learning resource and as a starting point for further product development.

1.3 Overview of the components

The application consists of the following core components:

• ABase62-Encoder , which transforms consecutive IDs into URL-compatible short forms
• AMapping-Store , which manages the mapping between the short link and the original URL
• AREST service , which allows URL shortening and resolution via redirect
• Oneoptional UI based on Vaadin Flow for manual management of mappings
  and aconfigurable WAR deployment that integrates all components

The architecture follows the principle:“As little as possible, as much as necessary.” Each part of the application is modular and allows for later splitting if necessary – for example, into separate services for reading, writing or analysis.

In the next chapter, we will focus on the concrete project structure and the module structure.

2. Project structure and module organisation

2.1 Structure of a modular WAR project

The first executable version of the URL shortener is realised as a monolithic Java application, which is in the form of a classic WAR.The project’s structure is based on a clear,layered architecture , which is prepared for later decomposition. The project is organised modularly, distinguishing between core logic, HTTP interface, and user interface. This separation not only allows for better maintainability but also forms the basis for the service decomposition planned in Part III or IV.

The project consists of three main modules:

• shortener-core: Contains all business logic, including URL encoding, data model and store interfaces.
• shortener-api: Implements the REST API based on the Java HTTP server (com.sun.net.httpserver.HttpServer).
• shortener-ui-required: Optional UI module with Vaadin Flow for managing and visualising mappings.

These modules are distributed via a central WAR project (shortener war), which handles the delivery configuration and combines all dependencies. The WAR project is the only one that handles servlet-specific aspects (e.g., web.xml, I require a Servlet) – the remaining modules remain entirely independent of it.

2.2 Separation of domain, API and UI code

2.2 Separation of Domain, API, and UI Code

The modularisation of the project is based on the principle of technological isolation: The core business logic must know nothing about HTTP, servlet containers, or UI frameworks. This way, it remains fully testable, interchangeable, and reusable—for example, for future CLI or event-based variants of the shortener.

The core module defines all central interfaces (UrlMappingStore, ShortCodeEncoder) as well as the base classes (ShortUrlMapping, Base62Encoder). These components do not contain any I/O logic.

The api module is responsible for parsing HTTP requests, routing, and generating redirects and JSON responses. It accesses the core logic internally but remains detached from UI aspects.

The ui-vaadin module uses Vaadin Flow to implement a web-based interface, integrates the core logic directly, and is initialised in the WAR via a dedicated servlet definition.

Additional modules can be optionally added—for example, for persistence, monitoring, or analysis—without compromising the coherence of the structure.

2.3 Tooling und Build (Maven, JDK 24, WAR-Plugin)

The build system is based on the current version of Maven. Each module is managed as a standalone Maven project with its pom.xml, with shortener-war configured as the parent WAR application. WAR packaging is handled using the standard servlet model, allowing the resulting file to be easily deployed in Tomcat, Jetty, or any Servlet 4.0+ compatible container.

Java 24 is required at runtime, which is particularly relevant for modern language features such as record, pattern matching, and SequencedMap. Release 21 or higher is recommended as the target platform to ensure compatibility with modern runtimes.

Vaadin Flow integration is handled purely on the server side via the Vaadin Servlet and does not require a separate front-end build pipeline. Resources such as themes and icons are loaded entirely from the classpath.

3. URL encoding: Base62 and ID generation

3.1 Design of a stable short link scheme

The key requirement for a URL shortener is to generate unique, shortest possible character strings that serve as keys for accessing the original URL. To meet this requirement, the first implementation utilises a sequential ID scheme that assigns a consecutive numeric ID to each new URL. This ID is then converted into a URL-compatible format—specifically, Base62.

Base62 includes the 26 uppercase letters, the 26 lowercase letters, and the 10 decimal digits. Unlike Base64, Base62 does not contain special characters such as +, /, or =, making it ideal for URLs: The generated strings are readable, system-friendly, and easily transferable in all contexts.

The resulting scheme is thus based on a two-step process:

• Assigning a unique numeric ID (e.g., 1, 2, 3, …)
• Converting this ID to a Base62 string (e.g., 1 → b, 2 → c, …)

This method guarantees unique and unguessable codes, especially if the ID count does not start at 0 or if codes are additionally shuffled.

3.2 Implementation of a Base62 encoder

TheBase62-Encoder is used as a standalone utility class in the core module. It contains two static methods:

• encode(long value): converts a positive integer to a Base62 string
• decode(String input): converts a Base62 string back to an integer

The alphabet is defined internally as a constant character string, and the conversion process is carried out purely mathematically, comparable to the representation of a number in another place value system.

This implementation creates a deterministic, stable, and thread-safe encoder that requires no external libraries. The resulting codes are significantly shorter than the underlying decimal number and contain no special characters—a key advantage for embedded or typed links.

For exceptional cases—such as custom aliases—the encoder remains optional, as such aliases can be stored directly as separate strings. However, by default, the Base62 encoder is the preferred method.

3.3 Alternatives: Random, Hashing, Custom Aliases

In addition to the sequential approach, there are other methods for generating short links that can be considered in later stages of development:

• Random-based tokens (z. B. UUID, SecureRandom) increase unpredictability, but require collision detection and additional memory overhead.
• The hashing process (e.g., SHA-1 of the destination URL) guarantees stability but is prone to collisions under high load or identical destination addresses.
• Custom aliases enable readable, short links (e.g., /helloMax), but require additional checking for collisions, syntactic validity, and protection of reserved terms.

For the first version, we focus on the sequential model with Base62 transformation – a stable and straightforward approach.

3.4 Implementation: Base62Encoder.java

The goal is to provide a simple utility class that converts integers to Base62 strings and vice versa. This class is thread-safe, stateless, and implemented without any external dependencies.

First, the complete source code:

javaCopy

publicfinalclassBase62Encoder{ privatestaticfinalStringALPHABET="0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"; privatestaticfinalintBASE=ALPHABET.length(); privateBase62Encoder(){ } publicstaticStringencode(longnumber){   if(number<0){     thrownewIllegalArgumentException("Only non-negative values supported");   }   StringBuilderresult=newStringBuilder();   do{     intremainder=(int)(number%BASE);     result.insert(0,ALPHABET.charAt(reminder));     number=number/BASE;   }while(number>0);   returnresult.toString(); } publicstaticlongdecode(Stringinput){   if(input==null||input.isEmpty()){     thrownewIllegalArgumentException("Input must not be null or empty");   }   longresult=0;   for(charc:input.toCharArray()){     intindex=ALPHABET.indexOf(c);     if(index==-1){       thrownewIllegalArgumentException("Invalid character in Base62 string: "+c);     }     result=result*BASE+index;   }   returnresult; }}

3.5 What is happening here – and why?

This class encapsulates all Base62 encoding behaviour in two static methods. The character set consists of digits (0–9), lowercase letters (a–z), and uppercase letters (A–Z), resulting in exactly 62 different characters.

• The method encode(long number) converts an integer into its inverse representation in the base 62 place value system. The remainder of the division by 62 is successively calculated, and the corresponding character is inserted. The result is a short, URL-friendly string.

• The decode(String input) method reverses this process: it converts a Base62 string back into its numeric representation. Each character is replaced by its index in the alphabet and weighted accordingly.

This implementation is robust against invalid input, operates entirely in memory, and can be used directly in ID generation or URL mappings.

Implementation: ShortCodeGenerator.java

The class abstracts ID generation from the concrete storage mechanism. It is suitable for both the in-memory version and future persistent variants. The generator is thread-safe and uses only JDK resources.

javaCopy

publicfinalclassShortCodeGenerator{ privatefinalAtomicLongcounter; publicShortCodeGenerator(longinitialValue){   this.counter=newAtomicLong(initialValue); } publicStringnextCode(){   longid=counter.getAndIncrement();   returnBase62Encoder.encode(id); } publiclongcurrentId(){   returncounter.get(); }}

---

Explanation

This class encapsulates a sequential counter that is incremented each time nextCode() generates a new, unique short code. The output is based on a monotonically increasing ID encoded into a Base62 string.

The getAndIncrement() method of AtomicLong isnon-blocking and thus highly performant, even under high concurrency conditions. The generated code is unambiguous, compact, and deterministic—properties that are ideal for auditing, logging, and subsequent analysis.

The constructor allows you to configure the initial value. This is useful, for example, if you want to continue a persistent counter after a restart.

Example usage (e.g. in the mapping store)

javaCopy

ShortCodeGeneratorgenerator=newShortCodeGenerator(1L);StringshortCode=generator.nextCode();//z. B."b","c","d",...

4. Mapping Store: Storage of the mapping

4.1 Interface-Design: UrlMappingStore

The mapping store forms the heart of the URL shortener. It manages the mapping between a short code (e.g.kY7zD) and the corresponding target URL (https://example.org/foo/bar). At the same time, it takes control over multiple uses, expiration times, and potential aliases.

In the first stage of development, a purelyin-memory-based solution is used**.** This is fast, simple, and ideal for starting—even if it is lost upon reboot. Persistence is deliberately postponed to a later phase.

The store is abstracted via a simple interface. This interface allows for later substitution (e.g., with a file- or database-based version) without affecting the API or UI components.

4.2 In-memory implementation with ConcurrentHashMap

The first concrete implementation utilises a ConcurrentHashMap to ensure reliable access even under heavy load. Each mapping entry is represented by a simple record object (ShortUrlMapping) that contains the target URL, creation time, and optional expiration information.

The combination of ConcurrentHashMap and ShortCodeGenerator allows deterministic and thread-safe ID assignment without the need for explicit synchronisation. This creates a high-performance solution that operates reliably even under high load.

4.3 Extensibility for later persistence

All entries are accessible via a central interface. This interface is used not only for storage and retrieval but also forms the basis for later extensions, such as a persistence layer with flat files or EclipseStore, process control via TTL, or even event-driven backends.

The data structure can be extended with metrics, validity logic, or audit fields without requiring changes to the API – a classic approach to interface orientation in the sense of the open/closed principles.

4.4 Implementation

xmlCopy

public record ShortUrlMapping(   String shortCode,   String originalUrl,   Instant createdAt,   Optional<Instant> expiresAt) {}

This structure represents the basic assignment and allows the optional specification of an expiration time.

Store-Interface: UrlMappingStore.java

xmlCopy

public interface UrlMappingStore { ShortUrlMapping createMapping(String originalUrl); Optional<ShortUrlMapping> findByShortCode(String shortCode); boolean exists(String shortCode); List<ShortUrlMapping> findAll(); boolean delete(String shortCode); int mappingCount();}

The interface is deliberately kept slim and abstracts the two core operations: insert (with creation) and lookup.

Implementation: InMemoryUrlMappingStore.java

xmlCopy

public class InMemoryUrlMappingStore   implements UrlMappingStore, HasLogger { private final ConcurrentHashMap<String,ShortUrlMapping> store                     = new ConcurrentHashMap<>(); private final ShortCodeGenerator generator; public InMemoryUrlMappingStore() {   this.generator = new ShortCodeGenerator(1L); } @Override public ShortUrlMapping createMapping(String originalUrl) {   logger().info("originalUrl: {} ->", originalUrl);   String code = generator.nextCode();   ShortUrlMapping shortMapping = new ShortUrlMapping(       code,       originalUrl,       Instant.now(),       Optional.empty()   );   store.put(code, shortMapping);   return  shortMapping; } @Override public Optional<ShortUrlMapping> findByShortCode(String shortCode) {   return Optional.ofNullable(store.get(shortCode)); } @Override public boolean exists(String shortCode) {   return store.containsKey(shortCode); } @Override public List<ShortUrlMapping> findAll() {   return new ArrayList<>(store.values()); } @Override public boolean delete(String shortCode) {   return store.remove(shortCode) != null; } @Override public int mappingCount() {   return store.size(); }}

4.5 Why so?

The use of a ConcurrentHashMap ensures that concurrent write and read operations can be handled consistently and efficiently. The combination with AtomicLong in ShortCodeGenerator prevents collisions. The interface allows for a persistent implementation to be introduced later without changing the API or UI behaviour.

5. HTTP API with Java tools

5.1 HTTP-Server mit com.sun.net.httpserver.HttpServer

Instead of relying on heavyweight frameworks like Spring or Jakarta EE, we use thelightweight HTTP server implementation that the JDK already includes in the package com.sun.net.httpserver. This API, although rudimentary, is performant, stable, and perfectly sufficient for our use case.

The server is configured in just a few lines, requires no XML or annotation-based mappings, and can be controlled entirely programmatically. For each path, we define a separate HTTP handler that receives the request, processes it, and returns a structured HTTP response.

5.2 POST /shorten: Shorten URL

The first endpoint allows a long URL to be passed over anHTTP POST to the shortener. In response, the server returns the generated short form, in the simplest case as a JSON object with the shortCode.

Example request:

POST /shorten

Content-Type: application/json

jsonCopy

{  "url":"https://example.com/some/very/long/path"}

Answer:

200 OK

Content-Type: application/json

jsonCopy

{  "shortCode":"kY7zD"}

Missing or invalid entries are marked with 400 Bad Request answered.

5.3 GET /{code}: Redirect to the original URL

When calling a shortcode (e.g.GET /kY7zD), the server checks whether a mapping exists. If so, a HTTP 302 redirect to the original address. If the code is unknown or expired, a 404 Not Found error will be displayed.

This redirection is stateless and allows for later isolation into a read-only redirect service.

5.4 Implementation

Starting point: ShortenerServer.java

javaCopy

publicclassShortenerServer   implementsHasLogger{ privateHttpServerserver; publicstaticvoidmain(String[]args)     throwsIOException{   newShortenerServer().init(); } publicvoidheat()     throwsIOException{   ourstore=newInMemoryUrlMappingStore();   this.server=HttpServer.create(newInetSocketAddress(8080),0);   server.createContext("/shorten",newShortenHandler(store));   server.createContext("/",newRedirectHandler(store));   server.setExecutor(null);// default executor   server.start();   System.out.println("URL Shortener server running at http://localhost:8080"); } publicvoidshutdown(){   if(server!=null){     server.stop(0);     System.out.println("URL Shortener server stopped");   } }}

---

POST Handler: ShortenHandler.java

javaCopy

publicclassShortenHandler   implementsHttpHandler,HasLogger{ privatefinalUrlMappingStorestore; publicShortenHandler(UrlMappingStorestore){   this.store=store; } @Override publicvoidhandle(HttpExchangeexchange)     throwsIOException{   if(!"POST".equalsIgnoreCase(exchange.getRequestMethod())){     exchange.sendResponseHeaders(405,-1);     return;   }   InputStreambody=exchange.getRequestBody();   Map<String,String>payload=parseJson(body);   StringoriginalUrl=payload.get("url");   logger().info("Received request to shorten url: {}",originalUrl);   if(originalUrl==null||originalUrl.isBlank()){     exchange.sendResponseHeaders(400,-1);     return;   }   ShortUrlMappingmapping=store.createMapping(originalUrl);   logger().info("Created mapping for {} -> {}",originalUrl,mapping.shortCode());   byte[]response=toJson(Map.of("shortCode",mapping.shortCode())).getBytes(StandardCharsets.UTF_8);   exchange.getResponseHeaders().add("Content-Type","application/json");   exchange.sendResponseHeaders(200,response.length);   try(OutputStreamos=exchange.getResponseBody()){     os.write(response);   } }}

GET-Handler: RedirectHandler.java

xmlCopy

public class RedirectHandler   implements HttpHandler , HasLogger { private final UrlMappingStorestore; public RedirectHandler(UrlMappingStore store) {   this.store = store; } @Override public void handle(HttpExchange exchange)     throws IOException {   our requestURI = exchange.getRequestURI();   our fullPath = requestURI.getPath();   logger().info("Full path: {}", fullPath);   String path = fullPath.substring(1); // strip leading '/'   logger().info("Path: {}", path);   if (path.isEmpty()) {     exchange.sendResponseHeaders(400, -1);     return;   }   Optional<String> target = store       .findByShortCode(path)       .map(ShortUrlMapping::originalUrl);   if (target.isPresent()) {     exchange.getResponseHeaders().add("Location", target.get());     exchange.sendResponseHeaders(302, -1);   } else {     exchange.sendResponseHeaders(404, -1);   } }}

JsonUtils.java (Minimal Java JSON without external libraries)

Since we do not want to use external dependencies such as Jackson or Gson in this first implementation, we need ourown utility class to process simple JSON objects, specifically:

• String → Map<String, String>: for processing HTTP POST payloads (/shorten)
• Map<String, String> → JSON-String: to generate responses (e.g.{ “shortCode”: “abc123” })

This class is sufficient for simple key-value structures, as used in the shortener. It isnot intended for nested objects or arrays , but as a pragmatic solution for the start.

Implementation: JsonUtils.java

javaCopy

publicfinalclassJsonUtils{ privateJsonUtils(){} publicstaticMap<String,String>parseJson(InputStreaminput)     throwsIOException{   Stringjson=readInputStream(input).trim();   returnparseJson(json); } @NotNull publicstaticMap<String,String>parseJson(Stringjson)     throwsIOException{   if(!json.startsWith("{")||!json.endsWith("}")){     thrownewIOException("Invalid JSON object");   }   Map<String,String>result=newHashMap<>();   // Remove curly braces   Stringbody=json.substring(1,json.length()-1).trim();   if(body.isEmpty()){     returnCollections.emptyMap();   }   // Separate key-value pairs with commas   String[]entries=body.split(",");   Arrays.stream(entries)       .map(entry->entry.split(":",2))       .filter(parts->parts.length==2)       .forEachOrdered(parts->{         Stringkey=unquote(parts[0].trim());         Stringvalue=unquote(parts[1].trim());         result.put(key,value);       });   returnresult; } publicstaticStringtoJson(Map<String,String>map){   StringBuildersb=newStringBuilder();   sb.append("{");   booleanfirst=true;   for(Map.Entry<String,String>entry:map.entrySet()){     if(!first){       sb.append(",");     }     sb.append("\"").append(escape(entry.getKey())).append("\":");     sb.append("\"").append(escape(entry.getValue())).append("\"");     first=false;   }   sb.append("}");   returnsb.toString(); } privatestaticStringreadInputStream(InputStreaminput)     throwsIOException{   try(BufferedReaderreader=newBufferedReader(       newInputStreamReader(input,StandardCharsets.UTF_8))){     returnreader.lines().collect(joining());   } } privatestaticStringunquote(Strings){   if(s.startsWith("\"")&&s.endsWith("\"")&&s.length()>=2){     returns.substring(1,s.length()-1);   }   returns; } privatestaticStringescape(Strings){   // simple escape logic for quotes   returns.replace("\"","\\\""); }}

Properties and limitations

This implementation is:

• fully JDK-based(no third-party libraries)
• forflat JSON objects suitable, i.e.{ “key”: “value” }
• robust against trivial parsing errors , but without JSON Schema validation
• Consciouslyminimalistic to stay within the scope of the prototype

It is sufficient for:

• POST /shorten(Client sends{ “url”: “…” })
• Response to this POST (server sends{ “shortCode”: “…” })

Example use

javaCopy

InputStreambody=exchange.getRequestBody();Map<String,String>input=JsonUtils.parseJson(body);StringshortCode="abc123";Stringresponse=JsonUtils.toJson(Map.of("shortCode",shortCode));

For productive systems, the following is recommended in the future:

• Gson(lightweight, idiomatic)
• Jackson(extensive, also for DTO binding)
• Json-B(Standard-API, Jakarta conform)

However, for our first implementation in the Core JDK, the solution shown above deliberately remains the appropriate middle ground.

5.6 Core Java Client Implementation

The class URLShortenerClient functions as a minimalist HTTP client for interacting with a URL shortener service. Its structure allows connection to a configurable or locally running server, with the default address beinghttp://localhost:8080/. This enables easy integration into local development environments, test runs, or automated system tests without the need for additional configuration.

At the heart of the functionality is the method shortenURL(String originalUrl). It initiates an HTTP POST call against the server endpoint/shorten, transmits the URL to be shortened in a simple JSON document and immediately evaluates the server’s response. Successful completion is indicated exclusively by a status code.200 OK In this case, the method extracts the contained shortCode using the static auxiliary method extractShortCode() from the class JsonUtils. If the server returns a different HTTP code instead, the process will be aborted with a corresponding IOException, which enforces explicit error handling at the application level. This maintains a clear semantic separation between regular usage and exception situations.

The second central method,resolveShortcode(String shortCode), is used to explicitly resolve a short URL. It sends a GET request directly to the server’s root context, supplemented by the passed code. The behaviour of this method largely corresponds to that of a web browser, with the difference that automatic redirects have been deliberately deactivated. This way, the method can determine the actual target address, if present, ​​from the HTTP header field. Location and return it as a result. It clearly distinguishes between valid redirects (status 301 or 302), non-existent codes (status 404), and other unexpected responses. In the latter case, an IOException is thrown, analogous to the shortening process.

Technically speaking, the URLShortenerClient is exclusively composed of the Java SE API, namely HttpURLConnection, TYPE and stream-based input and output routines. All communication is UTF-8 encoded, ensuring high interoperability with modern JSON-based REST interfaces. The class also implements the interface HasLogger, which suggests a project-wide logging infrastructure and implicitly supports good traceability in server communication.

This client is particularly recommended for integration tests, command-line tools, or administrative scripts that require specific URLs to be shortened or verified. Due to its lean structure, the class is also suitable as a starting point for further abstractions, such as service-oriented encapsulation in larger architectures.

javaCopy

publicclassURLShortenerClient   implementsHasLogger{ protectedstaticfinalStringDEFAULT_SERVER_PORT="8080"; protectedstaticfinalStringDEFAULT_SERVER_URL="http://localhost:"+DEFAULT_SERVER_PORT; protectedstaticfinalStringSHORTEN_URL_ENDPOINT="/shorten"; protectedstaticfinalStringREDIRECT_URL_ENDPOINT="/"; privatefinalTYPEserverBase; publicURLShortenerClient(StringserverBaseUrl){   this.serverBase=TYPE.create(serverBaseUrl.endsWith("/")?serverBaseUrl:serverBaseUrl+"/"); } publicURLShortenerClient(){   this.serverBase=TYPE.create(DEFAULT_SERVER_URL); } /**  * String originalUrl = "https://svenruppert.com";  *  * @param originalUrl  * @return  * @throws IOException  */ publicStringshortenURL(StringoriginalUrl)     throwsIOException{   ourserverURL=serverBase.toURL();   // --- Step 1: POST to the /shorten endpoint with a valid URL ---   URLshortenUrl=URI.create(serverURL+SHORTEN_URL_ENDPOINT).toURL();   HttpURLConnectionconnection=(HttpURLConnection)shortenUrl.openConnection();   connection.setRequestMethod("POST");   connection.setDoOutput(true);   connection.setRequestProperty("Content-Type","application/json");   Stringbody="{\"url\":\""+originalUrl+"\"}";   try(OutputStreamos=connection.getOutputStream()){     os.write(body.getBytes());   }   intstatus=connection.getResponseCode();   if(status==200){     try(InputStreamis=connection.getInputStream()){       StringjsonResponse=newString(is.readAllBytes(),UTF_8);       StringextractedShortCode=JsonUtils.extractShortCode(jsonResponse);       logger().info("extractedShortCode .. {}",extractedShortCode);       returnextractedShortCode;     }   }else{     thrownewIOException("Server returned status "+status);   } } publicStringresolveShortcode(StringshortCode)     throwsIOException{   logger().info("Resolving shortCode: {}",shortCode);   URLurl=URI.create(DEFAULT_SERVER_URL+REDIRECT_URL_ENDPOINT+shortCode).toURL();   logger().info("url .. {}",url);   HttpURLConnectionconnection=(HttpURLConnection)url.openConnection();   connection.setInstanceFollowRedirects(false);   intresponseCode=connection.getResponseCode();   logger().info("responseCode .. {}",responseCode);   if(responseCode==302||responseCode==301){     ourlocation=connection.getHeaderField("Location");     logger().info("location .. {}",location);     returnlocation;   }elseif(responseCode==404){     returnnull;   }else{     thrownewIOException("Unexpected response: "+responseCode);   } }}

5.7 Summary

With just a few lines, you can create a functional HTTP API that serves as an ideal testbed and proof of concept. The structure is minimal but open to extensions such as error objects, rate limiting, or logging. What’s particularly remarkable is that the entire API works without servlet containers, external frameworks, or reflection—ideal for embedded applications and lightweight deployments.

In the next blog post, we will create a graphical interface to map user interactions.

Happy Coding

Sven

]]>Javahttps://svenruppert.com/posts/short-links-clear-architecture-a-url-shortener-in-core-java/Tue, 10 Jun 2025 22:43:22 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/short-links-clear-architecture-a-url-shortener-in-core-java/A URL shortener seems harmless – but if implemented incorrectly, it opens the door to phishing, enumeration, and data leakage. In this first part, I’ll explore the theoretical and security-relevant fundamentals of a URL shortener in Java – without any frameworks, but with a focus on entropy, collision tolerance, rate limiting, validity logic, and digital responsibility. The second part covers the complete implementation: modular, transparent, and as secure as possible.<![CDATA[

A URL shortener seems harmless – but if implemented incorrectly, it opens the door to phishing, enumeration, and data leakage. In this first part, I’ll explore the theoretical and security-relevant fundamentals of a URL shortener in Java – without any frameworks, but with a focus on entropy, collision tolerance, rate limiting, validity logic, and digital responsibility. The second part covers the complete implementation: modular, transparent, and as secure as possible.

1. 1.1 Motivation and use cases
2. 1.2 Differentiation from related technologies
3. 1.3 Objective of the paper
4. 2.1 URI, URL and URN – conceptual basics
5. 2.2 Principles of address shortening
6. 2.3 Entropy, collisions and permutation spaces
7. 4.2 URL Encoding: Hashing, Base62 and Alternatives
8. 4.3 Mapping Store: Interface, Implementation, Synchronisation
9. 4.4 REST API with pure Java (HTTP server, handler, routing)
10. 4.5 Error handling, logging and monitoring
11. 5.1 Abuse opportunities and protection mechanisms
12. 5.2 Rate limiting and IP-based throttling
13. 5.3 Validity period and deletion concepts
14. 5.4 Protection against enumeration and information leakage
15. 6.1 Access times and hash lookups
16. 6.2 Memory usage and garbage collection
17. 6.3 Benchmarking: Local tests and load simulation
18. 7.1 Custom aliases
19. 7.2 Access counting and analytics
20. 7.3 QR-Code-Integration
21. 7.4 Integration into messaging or tracking systems
22. 8.1 Data protection for link tracking
23. 8.2 Responsibility for forwarding
24. 8.3 Transparency and disclosure of the destination address
25. 9.1 Lessons Learned
26. 9.2 Possible further developments (e.g. blockchain, DNSSEC)
27. 9.3 Importance of URL shorteners in the context of digital sovereignty

1.1 Motivation and use cases

In an increasingly fragmented and mobile information world, URLs are not just technical addressing mechanisms; they are central building blocks of digital communication. Long and hard-to-remember URLs are a hindrance in social media, emails, or QR codes, as they are not only aesthetically unappealing but also prone to errors when manually entered. URL shorteners address this problem by generating compact representations that point to the original target address. In addition to improved readability, aspects such as statistical analysis, access control, and campaign tracking also play a key role.

Initially popularised by services like TinyURL or bit.ly, URL shorteners have now become integrated into many technical infrastructures – from marketing platforms and messaging systems to IoT applications, where storage and bandwidth restrictions play a significant role. A shortened representation of URLs is also a clear advantage in the context of QR codes or limited character sets (e.g., in SMS or NFC data sets).

1.2 Differentiation from related technologies

A URL shortener is not a classic forwarding platform and is conceptually different from proxy systems, link resolvers, or load balancers. While the latter often operate at the transport or application layer (Layer 4 or Layer 7 in the OSI model) and optimise transparency, availability, or performance, a shortener primarily pursues the goal of simplifying the display and management of URLs. Nevertheless, there are overlaps, particularly in the analysis of access patterns and the configuration of redirect policies.

In this work, a minimalist URL shortener is designed and implemented. It deliberately avoids external frameworks to implement the central concepts in a comprehensible and transparent manner in Core Java. The choice of Java 24 enables the integration of modern language features, such as records, sealed types, and virtual threads, into a secure and robust architecture.

1.3 Objective of the paper

This paper serves a dual purpose: on the one hand, it aims to provide a deep technical understanding of the functionality and challenges associated with a URL shortener. On the other hand, it serves as a practical guide for implementing such a service using pure Java—that is, without Spring, Jakarta EE, or external libraries.

To this end, a comprehensive architecture will be developed, implemented, and continually enhanced with key aspects such as security, performance, and extensibility. The focus is deliberately on a system-level analysis of the processes to provide developers with a deeper understanding of the interaction between the network layer, coding strategies, and persistent storage. The goal is to develop a viable model that can be utilised in both educational contexts and as a basis for productive services.

2. Technical background

2.1 URI, URL and URN – conceptual basics

In everyday language, terms such as “URL” and “link” are often used synonymously, although in a technical sense, they describe different concepts.URI (Uniform Resource Identifier) refers to any character string that can uniquely name or locate a resource. A URL**(Uniform Resource Locator)** is a special form of a URI that not only identifies but also describes the access path, for example, through a protocol such as https, ftp, or mailto. AURN (Uniform Resource Name), on the other hand, names a resource persistently without referring to its physical address, such as urn:isbn:978-3-16-148410-0.

In the context of URL shorteners, URLs are exclusively concerned with accessible paths, typically via HTTP or HTTPS. The challenge is to transform these access paths in a way that preserves their semantics while reducing their representation.

2.2 Principles of address shortening

The core idea of ​​a URL shortener is to replace a long URL string with a shorter key that points to the original address via a mapping. This mapping is done either directly in a lookup store (e.g., hash map, database table) or indirectly via a computational method (e.g., a hash function with collision management).

The goal is to use the redundancy of long URLs to map their entropy to a significantly shorter string. This poses a trade-off between collision-freeness, brevity, and readability. Conventional methods are based on encoding unique keys in a Base62 alphabet ([0-9a-zA-Z]), which offers 62 states per character. Just six characters can represent over 56 billion unique URLs—sufficient for many productive applications.

The shortcode acts as the primary key for address resolution. It is crucial that it is stable, efficiently generated, and as challenging to guess as possible to prevent misuse (e.g., brute-force enumeration).

2.3 Entropy, collisions and permutation spaces

A key aspect of URL shortening is the question of how many different short addresses a system can actually generate. This consideration directly depends on the length of the generated shortcuts and their character set. Many URL shorteners use a so-called Base62 alphabet. This includes the ten digits from zero to nine, the 26 lowercase letters, and the 26 uppercase letters, for a total of 62 different characters.

For example, if you generate abbreviations with a fixed length of six characters, you get a combinatorial space in which over 56 billion different character strings are possible. Even with this relatively short number of characters, billions of unique URLs can be represented, which is more than sufficient for many real-world applications. For longer abbreviations, the address space grows exponentially.

But the sheer number of possible combinations is only one aspect. How these shortcuts are generated is equally important. If the generation is random, it is essential to ensure that no duplicate codes are created – so-called collisions. These can be managed either by checking for their existence beforehand or by deterministic methods such as hash functions. However, hash methods are not without risks, especially under heavy load: The more entries there are, the higher the probability that two different URLs will receive the same short code, especially if the hash function has not been optimised for this use case.

Another criterion is the distribution of the generated shortcuts. A uniform distribution in the address space is desirable because, on the one hand, it reduces the risk of collisions, and on the other hand, it increases the efficiency of storage and retrieval mechanisms – for example, in sharding for distributed systems or caching in high-traffic environments. Cryptographically secure random numbers or specially designed generators play a crucial role here.

Overall, it can be said that the choice of alphabet, the length of the abbreviations and the way they are generated are not just technical parameters, but fundamental design decisions that significantly influence the security, efficiency and scalability of a URL shortener.

3. Architecture of a URL shortener

The architecture of a URL shortener is surprisingly compact at its core, but by no means trivial. Although its basic function is simply to link a long URL with a short alias, numerous technical and conceptual decisions arise in the details. These include data storage, the structure of API access, concurrency behaviour, and security against misuse. This chapter explains the central components and their interaction, deliberately avoiding external frameworks. Instead, the focus is on a modular, transparent structure in pure Java.

At the heart of the system is a mapping table – typically in the form of a map or a persistent key-value database – that uniquely assigns each generated short code to its corresponding original URL. This structure forms the backbone of the shortener. Crucially, this mapping must be both efficiently readable and consistently modifiable, especially under load or when accessed concurrently by multiple clients.

A typical URL shortener consists of three logically separate units: an input endpoint for registering a new URL, a redirection endpoint for evaluating a short link, and a management unit that provides metadata such as expiration times or access counters. In a purely Java-based solution without frameworks, network access is provided via the HTTP server introduced in Java 18.com.sun.net.httpserver package. This allows you to define REST-like endpoints with minimal overhead and to communicate withHttpExchange objects.

There are various options for storing mappings. In-memory structures, such asConcurrentHashMap , offer maximum speed but are volatile and unsuitable for productive applications without a backup mechanism. Alternatively, file-based formats, relational databases, or object-oriented stores such as EclipseStore can be used. This paper will initially work with volatile storage to illustrate the basic logic. Persistence will be added modularly later.

Another key aspect concerns concurrency behaviour. Since URL shorteners are typically burdened by a large number of read accesses, for example, when calling short links, the architecture must be designed to allow concurrent access to the lookup table without locking conflicts. The same applies to the generation of new shortcuts, which must be atomic and collision-free. Java 24 introduces modern language tools, including virtual threads and structured concurrency, which can be utilised to manage server load in a more deterministic and scalable manner.

Last but not least, horizontal extensibility plays a role. A cleanly decoupled design allows the shortener to be easily transferred to distributed systems later. For example, the actual URL resolver can be operated as a stateless service, while data storage is outsourced to a shared backend. Caching strategies and load balancing can also be integrated much more easily in such a setup.

In summary, a URL shortener is much more than a simple string replacement. Its architecture must be both efficient, robust, and extensible—properties that can be easily achieved through a modular structure in pure Java.

4. Implementation with Java 24

4.1 Project structure and module overview

The implementation of the URL shortener follows a modular structure that supports both clarity in the source code and testability, as well as extensibility. The project is structured as a Java module and leverages the capabilities of the Java Platform Module System (JPMS). The goal is to separate the core functionality—that is, the management of URL mappings—from the network layer and persistence. This keeps the business logic independent of specific storage or transport mechanisms.

At the centre is a module calledshortener.core , which contains all domain-specific classes: for example, the ShortUrlMapping, the UrlEncoder, as well as the centralUrlMappingStore interface with a simple implementation in memory. A moduleshortener.http , which is based on Java’s internal HTTP server. It implements the REST endpoints and utilises the core module’s components for actual processing. Additional optional modules, such as those for persistence or analysis, can be added later.

To organise the code, a directory structure that clearly reflects the module and layer boundaries is recommended. Within the modules, a distinction should be made betweenapi ,impl ,util and, if necessary,service.

4.2 URL Encoding: Hashing, Base62 and Alternatives

A central element of the shortener is the mechanism for generating short, unique codes. This implementation uses a hybrid method that generates a consecutive, atomic sequence number and converts it into a human-readable format using a Base62 encoder.

This choice has two advantages: First, it is deterministic and avoids collisions without the need for complex hash functions. Second, generated codes can be efficiently serialised and are easy to read, which is particularly relevant in marketing or print contexts. Alternatively, cryptographic hashes such as SHA-256 can be used when unpredictability and integrity protection are essential, for example, for signed links or zero-knowledge schemes.

The Base62 encoder is implemented as a pure utility class that encodes integer values ​​into a character string, where the alphabet consists of numbers and letters. Inverse decoding is also provided in case bidirectional analysis is required in the future.

4.3 Mapping Store: Interface, Implementation, Synchronisation

For managing URL mappings, a clearly defined interface calledUrlMappingStore provides methods for inserting new mappings, resolving short links, and optionally managing metadata. The default implementation, InMemoryUrlMappingStore, is based on a ConcurrentHashMap and utilisesAtomicLong for sequence number generation.

This simple architecture is completely thread-safe and allows parallel access without external synchronisation mechanisms. The implementation can be replaced at any time with a persistent variant, for example, based on flat file storage or through integration with an object-oriented storage system such as EclipseStore.

This separation keeps the application core stable while treating storage as a replaceable detail—a classic example of the dependency inversion principle in the spirit of Clean Architecture.

4.4 REST API with pure Java (HTTP server, handler, routing)

The REST interface is implemented exclusively with the built-in tools of the JDK. Java provides the packagecom.sun.net.httpserver , which offers a minimalistic yet powerful HTTP server ideal for lean services. For the implementation of the API, a separateHttpHandler is defined that responds to specific routes, such as/shorten for POST requests and/{code} for forwarding.

The implementation is based on a clear separation between parsing, processing, and response generation. Incoming JSON messages are parsed manually or with the help of simple helper classes, without the need for external libraries. HTTP responses also follow a minimalist format, characterised by structured status codes, simple header management, and UTF-8-encoded bodies.

Routing is handled by a dispatcher class, which selects the appropriate handler based on the request path and HTTP method. Later extensions, such as CORS, OPTIONS handling, or versioning, are easily possible.

4.5 Error handling, logging and monitoring

In a productive environment, robust error handling is essential. The implementation distinguishes between systematic errors (such as invalid inputs or missing short codes) and unexpected runtime errors (such as IO problems or race conditions). The former are reported with clear HTTP status codes, such as400 (Bad Request) or404 (Not Found). The latter leads to a generic500 Internal Server Erro r, with the causes being logged internally.

For logging, the JDK’s ownjava.util.logging This allows for platform-independent logging and can be replaced with SLF4J-compatible systems if needed. Monitoring metrics such as access counts, response times, or error statistics can be made accessible via a separate endpoint or JMX.

5. Security aspects

5.1 Abuse opportunities and protection mechanisms

A URL shortener can easily be used to obscure content. Attackers deliberately exploit the shortening to redirect recipients to phishing sites, malware hosts, or dubious content without the target address being immediately visible. This can pose significant risks, especially for automated distributions via social networks, chatbots, or email campaigns.

An adequate protection mechanism consists of automatically validating all target addresses upon insertion, for example, through syntactical URL checks, DNS resolution, and optionally through a background query (head request or proxy scan) that ensures that the target page is accessible and non-suspicious. Such checks should be modular so that they can be activated or deactivated depending on the environment (e.g., offline operation). Additionally, logging should be performed every time a short link is accessed, making it easier to identify patterns of abuse.

5.2 Rate limiting and IP-based throttling

Another risk lies in excessive use of the service, be it through botnets, targeted enumeration, or simple DoS behaviour. A robust URL shortener should therefore have rate limiting that restricts requests within a given time slot. This can be global, IP-based, or per-user, depending on the context.

In a Java implementation without frameworks, this can be achieved, for example, via aConcurrentHashMap that maintains a timestamp or counter buffer for each IP address. If a threshold is exceeded, the connection is terminated with a status code of429 Too Many Requests rejected. This simple throttling can be supplemented with leaky bucket or token bucket algorithms if necessary to achieve a fairer distribution over time. For productive use, logging of critical threshold violations is also recommended.

5.3 Validity period and deletion concepts

Not every short link should remain valid forever. A configurable validity period is essential, especially for security-critical applications, such as temporary document sharing or one-time authentication. A URL shortener should therefore offer the option of defining expiration times for each mapping.

On a technical level, it is sufficient to assign an expiration date to each mapping, which is checked during the lookup. When accessing expired short links, either an error status, such as410 Gone, is displayed, or the user is redirected to a defined information page. Additionally, there should be periodic cleanup mechanisms that remove expired or unused entries from memory, such as through a time-controlled cleanup process or lazy deletion upon access.

5.4 Protection against enumeration and information leakage

An often overlooked attack vector is the systematic scanning of the abbreviation space – for example, by automated retrieval of/aaaaaa until/zzzzzz. If a URL shortener delivers valid links without any protection mechanisms, potentially confidential information about the existence and use of links can be leaked.

An adequate protection consists in making the shortcuts themselves non-deterministic – for example, by using cryptographically generated, unpredictable tokens instead of continuous sequences. Additionally, access restrictions can be introduced, allowing only authenticated clients to access certain short links or excluding specific IP ranges. The targeted obfuscation of error responses – for example, by consistently issuing404 Not Found even with blocked or expired abbreviations – makes analysis more difficult for attackers.

A further risk arises when metadata such as creation time, number of accesses, or request origin is exposed unprotected via the API. Such information should only be accessible to authorised users or administrative interfaces and should never be part of the public API output.

6. Performance and optimisation

6.1 Access times and hash lookups

The most common operation in a URL shortener is resolving a shortcode into its corresponding original URL. Since this is a classic lookup operation, the choice of the underlying data structure is crucial. In the standard implementation, aConcurrentHashMap, which is optimised in Java 24, has fine-grained locking. This offers nearly constant access times – even under high concurrency – and is therefore ideal for read-intensive workloads, such as those typical of a shortener.

The latency of such an operation is in the range of a few microseconds, provided the lookup table is stored in main memory and no additional network or IO layers are involved. However, if data storage is outsourced to persistent systems, such as a relational database or a disk-based key-value store, the access time increases accordingly. Therefore, it is recommended to cache frequently accessed entries – either directly in memory or via a dedicated cache layer.

Performance also plays a role in the creation of new abbreviations. This is where sequence number generation usingAtomicLong is used, providing a thread-safe, low-contention solution for linear ID assignment. Combined with Base62 encoding, this creates a fast, predictable, and collision-free process.

6.2 Memory usage and garbage collection

Since a URL shortener must manage a growing number of entries over a longer period, it is worthwhile to examine its storage behaviour.ConcurrentHashMap. While this results in fast access times, it also means that all active mappings remain permanently in memory—unless cleanup is implemented. A simple mapping structure consisting of a shortcode, original URL, and an optional timestamp requires several hundred bytes per entry, depending on the JVM configuration and string length.

With several million entries, heap usage can reach several gigabytes. To improve efficiency, care should be taken to use objects sparingly. For example, common URL prefixes (e.g.https://) are replaced with symbolic constants. Records instead of classic POJOs also help reduce object size and minimise GC load.

In the long term, it is recommended to introduce an active or passive cleanup mechanism, such as TTL-based eviction or access counters, to specifically remove rarely used entries.WeakReference or soft caching should be considered with caution, since the semantics of such structures do not always lead to expected behaviour in the server context.

6.3 Benchmarking: Local tests and load simulation

Systematic benchmarking is essential for objectively evaluating the performance of a URL shortener. At a local level, this can be achieved with simple Java benchmarks that measure sequence number generation, lookup time, and code distribution quality. Tools such as JMH (Java Microbenchmark Harness) can also be used. Although external tools are not used in this paper, a manual microbenchmarking approach using System.nanoTime and a targeted warm-up can provide valuable insights.

For more realistic tests, a load simulation with HTTP clients is suitable, for example, using simple JDK-based multi-thread scripts or tools such ascurl. In particular, behaviour under high concurrent access load should be observed, both in terms of response times and resource consumption. Behaviour in the event of failed requests, rapid-fire access, or expired links should also be explicitly tested.

The goal of such benchmarks is not only to validate the maximum transaction rate, but also to verify stability under continuous load. A robust implementation should not only be high-performance but also deterministic in its response behaviour and resistant to out-of-memory errors. Optional profiling—for example, using JDK Flight Recorder—can reveal further optimisation potential.

7. Expansion options and variants

7.1 Custom aliases

A frequently expressed wish in practice is the ability to not only use automatically generated short links, but also to assign custom aliases – for example, for marketing campaigns, internal documents, or individual redirects. A custom alias, such as/travel2025 is much easier to remember than a random Base62 token and can be integrated explicitly into communication and branding.

Technically speaking, this expands the mapping store’s responsibility. Instead of only accepting numerically generated keys, the API must verify that a user-defined alias is syntactically valid, not already in use, and not reserved. A simple regex check, supplemented by a negative list for reserved terms (e.g./admin ,/api), is sufficient to get started. This alias must then be treated equally to the automatically generated codes when stored.

This creates new failure modes, for example, when a user requests an alias that already exists. Such cases should be handled consistently with a409 Conflict. The API can optionally suggest alternative names—a small convenience feature with a significant impact on the user experience (UX).

7.2 Access counting and analytics

A functional URL shortener is more than just a redirection tool—it’s also an analytics tool. Tracking how often, when, and from where a short link was accessed is particularly relevant in the context of campaigns, product pages, or documented distribution.

To implement this functionality, each successful resolution of a short link must be saved as an event, ​​either by simply incrementing a counter or by fully logging with a timestamp, IP address, and user agent. For the in-memory variant, an additionalAtomicLong or a metric structure aggregated via a map. Alternatively, detailed access data can be persisted in a dedicated log file or an external analytics module.

The evaluation can be performed either synchronously via API endpoints (e.g.,/stats/{alias}) or asynchronously via export formats such as JSON, CSV, or Prometheus metrics. Integration with existing logging systems (e.g. viajava.util.logging orLogstash) is easily possible.

7.3 QR-Code-Integration

For physical media, such as posters, packaging, or invitations, displaying a short link as a QR code is a useful extension. Integrating QR code generation into the URL shortener enables the direct generation of a visually encoded image of the link from the API.

Since no external libraries are used, QR code generation can be performed using a compact Java-based algorithm, such as one based on bit matrix generation and SVG output. Alternatively, a Base64-encoded PNG file can be delivered via an endpoint URL such as/qr/{alias}. The underlying data structure remains unchanged – only the representation is extended.

This feature not only enhances practical utility but also expands the service’s reach across multiple media channels.

7.4 Integration into messaging or tracking systems

In production architectures, a URL shortener typically operates in conjunction with other components. Instead, it is part of larger pipelines – for example, in email delivery, chatbots, content management systems, or user interaction tracking. Flexible integration with messaging systems such as Kafka, RabbitMQ, or simple webhooks allows every link creation or access to be transmitted as an event to external systems.

In a pure Java environment, this can be done via simple HTTP requests, log files, or asynchronous event queues. Scenarios are conceivable in which a notification is automatically sent to a third-party system for each new short link, for example, to generate personalised campaigns or for auditing purposes. Access to short links can also be mapped via events, which are subsequently statistically evaluated or visualised in dashboards.

Depending on the level of integration, it is recommended to implement a dedicated event dispatcher that encapsulates incoming or outgoing events and forwards them in a loosely coupled manner. This keeps the shortener itself lean and responsibilities clearly distributed.

8. Legal and ethical aspects

8.1 Data protection for link tracking

A URL shortener that logs visits automatically operates within the framework of data protection law. As soon as data such as IP addresses, timestamps, or user agents are stored, it is considered personal information in the legal sense, at least potentially. In the European Union, such data falls under the General Data Protection Regulation (GDPR), which entails specific obligations for operators.

The technical capability for analytics—for example, through access counting or geo-IP analysis—should therefore not be enabled implicitly. Instead, a URL shortener should be designed so that tracking mechanisms must be explicitly enabled, ideally with clear labelling for the end user. A differentiated configuration that distinguishes between anonymised and personal data collection is strongly recommended in professional environments.

Additionally, when storing personal data, a record of processing activities must be maintained, a legal basis (e.g., legitimate interest or consent) must be specified, and a defined retention period must be established. For publicly accessible shorteners, this may mean that tracking remains deactivated by default or is controlled via consent mechanisms. The implementation of such control structures is not part of the core functionality, but is an integral part of data protection-compliant operations.

8.2 Responsibility for forwarding

Another key point is the service provider’s responsibility for the content to which the link is redirected. Even if a shortener technically only implements a redirect, legal responsibility arises as soon as the impression arises that the operator endorses or controls the target content. This is especially true for public or embedded shorteners, such as those found in corporate portals or social platforms.

The challenge lies in distinguishing between technical neutrality and de facto mediation. It is therefore advisable to integrate legal protection mechanisms into the architecture, for example, through a policy that excludes the upload of specific domains, regular URL revalidation, or the use of abuse detection systems. In the event of misuse or complaints, immediate deactivation of individual mappings should be possible, ideally via a separate administration interface.

This responsibility is not only legally relevant but also has a reputational impact: Shorteners used to spread harmful content quickly lose their credibility – and possibly also their access to platforms or search engines.

8.3 Transparency and disclosure of the destination address

A common criticism of URL shorteners is that the destination address is no longer visible to the user. This limits the ability to evaluate whether a link is trustworthy before clicking on it. From an ethical perspective, this raises the question of whether a shortener should offer a pre-check option.

Technically, this can be achieved through a special preview mode, such as via an appendage, by explicitly calling an API or HTML preview page that transparently resolves the mapping, for example, a link likehttps://short.ly/abc123+. Instead of redirecting immediately, the user first displays an information page that displays the original URL and redirects to the page if desired. This function can be supplemented with information about validity, access statistics, or trustworthiness.

A transparent approach to redirects not only increases user acceptance but also reduces the potential for abuse, especially among security-conscious target groups. In sensitive environments, a mandatory preview page – for example, for all non-authenticated users – can be a helpful measure.

9. Conclusion and outlook

9.1 Lessons Learned

The development of a URL shortener in pure Java, without frameworks or external libraries, has demonstrated how even seemingly trivial web services, upon closer inspection, reveal themselves to be complex systems with diverse requirements. From the basic function of address shortening to security aspects and operational and legal implications, the result is a system that must be architecturally well-structured, yet flexible and extensible.

The importance of a clear separation of responsibilities is particularly important: A stable mapping store, a deterministic encoder, a secure yet straightforward REST API, and understandable error handling form the backbone of a robust service. Modern language tools from Java 24, such as records, sealed types, and virtual threads, enable a remarkably compact, type-safe, and concurrency-capable implementation.

The conscious decision against frameworks not only maximised the learning effect but also contributed to a deeper understanding of HTTP, data storage, thread safety, and API design – a valuable perspective for developers who want to operate in a technology-independent environment.

9.2 Possible further developments (e.g. blockchain, DNSSEC)

Despite their apparent simplicity, URL shorteners represent a fascinating field for technological innovation. There are efforts to move away from centralised management of the mapping between short code and target URL, instead using decentralised technologies such as blockchain. In this case, each link is stored as a transaction, providing resistance to manipulation and historical traceability. In practice, however, this places high demands on latency and infrastructure, which is why such approaches have been used so far rarely in production.

Another development strand lies in integration with DNSSEC-based procedures. This not only signs the shortcode itself, but also cryptographically verifies the authenticity of the resolved host. This could combine trust and verification, especially in security-critical areas such as government services, banks, or certificate authorities.

AI-supported heuristics, such as those for misuse detection or memory cleanup prioritisation, also offer potential. However, the integration of such mechanisms requires a data-efficient, explainable design that is compatible with applicable data protection regimes.

9.3 Importance of URL shorteners in the context of digital sovereignty

In today’s digital landscape, URL shorteners are more than just a convenience feature; they are a valuable tool. They influence the visibility, accessibility, and traceability of content. The question of whether and how a link is modified or redirected has a direct impact on information sovereignty and transparency, and thus on digital sovereignty.

Especially in the public sector, educational institutions, or organisations with strict compliance requirements, URL shorteners should not be operated as outsourced cloud services; instead, they should be developed in-house or at least integrated in a controlled manner. A self-hosted solution not only allows complete control over data flows and access histories but also protects against censorship-like outages or data-driven tracking by third parties.

This makes the URL shortener, as inconspicuous as its function may seem, a strategic component of a trustworthy IT infrastructure. It exemplifies the question: Who controls the path of information? In this respect, a custom shortener is not just a tool, but also a statement of identity.

The next part will be about the implementation itself..

Happy Coding

]]>JavaSecurityUncategorizedhttps://svenruppert.com/posts/if-hashcode-lies-and-equals-is-helpless/Fri, 06 Jun 2025 20:53:34 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/if-hashcode-lies-and-equals-is-helpless/A deep look into Java’s HashMap traps – visually demonstrated with Vaadin Flow. The silent danger in the standard library The use of HashMap and HashSet is a common practice in everyday Java development. These data structures offer excellent performance for lookup and insert operations, as long as their fundamental assumptions are met. One of them is hashCode() of a key remains stable. But what if that’s not the case?<![CDATA[

A deep look into Java’s HashMap traps – visually demonstrated with Vaadin Flow.

The silent danger in the standard library

The use ofHashMap andHashSet is a common practice in everyday Java development. These data structures offer excellent performance for lookup and insert operations, as long as their fundamental assumptions are met. One of them ishashCode() of a key remains stable. But what if that’s not the case?

This is precisely where one of the most subtle and dangerous traps of the Java standard library lurks: mutable key objects. In this article, we not only demonstrate why this constellation is problematic but also illustrate the phenomenon interactively using Vaadin Flow. Readers will learn how the HashMap works internally, whyequals() alone is not enough, and how to use modern language tools to generate robust, immutable keys.

1. The silent danger in the standard library
2. The fundamental problem: identity, hash codes and lookup
3. The classic mistake hashCode() depends on variable attributes
4. Security-critical consequences: When loss of consistency becomes an attack surface
5. Interactive Demo with Vaadin Flow
6. HashSet
7. Strategies to avoid
8. Why are other map implementations not affected
9. Conclusion: The price of convenience
10. Demo source code to try it out yourself

The fundamental problem: identity, hash codes and lookup

Internally, each HashMap is an array of buckets, where the hash code of the key determines the position of an entry. The insertion process (put(K key, V value)) looks like this: First, the map callskey.hashCode() and calculates the index in the bucket array from this value, using internal spreading. If entries already exist at this position (e.g., due to hash collisions), a linear search is performed within the bucket or – if there are enough collisions – i.e., if there are more than eight entries in a bucket and the underlying array exceeds a specific size (default value: 64) – the bucket is internally transformed from a simple linked list into a balanced binary tree structure (more precisely: a red-black tree). This conversion improves the lookup performance from linear time O(n) to logarithmic time O(log n). The decision for these thresholds is based on the observation that collisions are rare in well-chosen hash functions, and the overhead of a tree structure is only worthwhile with a high entry density.equals() is used to identify the appropriate key or create a new entry.

When accessing (get(Object key)) the same process occurs: The hash code of the transferred key is calculated again, the corresponding bucket is determined and searched for a matching key viaequals() However, if the hash code of the object differs from the originalput() have changed, a different bucket is addressed – the original entry then remains invisible.

Removing (remove(Object key)) is subject to the same mechanisms: the map searches for the correct bucket viahashCode() and then compares the keys usingequals(). Consistency of the hash-relevant data over the entire lifetime of the key is therefore essential. Only if this is guaranteed can the HashMap ensure its efficiency and correctness.

This means: Even access to the correct bucket depends exclusively on the result of the methodhashCode() This value is calculated immediately upon access, combined with an internal transformation (such as bitwise shifting and XOR operations to achieve a more even distribution across the bucket array), and then used to index the bucket array. If the return value ofhashCode(). After inserting an object into the map, for example, by mutating an attribute that is included in the calculation, the newly calculated index points to a different bucket. However, the object being searched for does not exist there, which is why the map can no longer find the entry. This behaviour is not a malfunction of the HashMap, but the direct consequence of a breach of the fundamental contracthashCode() must remain consistent while maintaining the same internal object state. Violating this contract is abusing the HashMap in a way it was not designed for, with potentially serious consequences for data integrity and program logic.

The classic mistake hashCode() depends on variable attributes

Let’s take the following example: An instance of the classPerson with the valuesname = “Alice” andid = 42 is created and stored as a key in a HashMap At the time of insertion, the map calculates thehashCode() based on the current state of the object – i.e.Objects.hash(“Alice”, 42) – and saves the entry in the corresponding bucket. After that, the field name of the object, e.g., is set to the value “Bob”. This changes the return value ofhashCode() , for example,Objects.hash(“Bob”, 42) , which addresses a different bucket.

If you now try to access the map again with the same object,map.get(originalPerson) – the operation fails. The map calculates a new index from the current hash code, looks in the corresponding bucket, and doesn’t find a matching entry. The original object is technically still in the map, but cannot be found.

The situation becomes even more misleading when one considers theentrySet() method**.** The entry is visible there, since the iteration is carried out directly via the internal chains, independent of the hash code. A comparison viaequals() works with a newly created, identical object – after all,equals() is typically based on content equality and not on memory address or hash code.

This makes the HashMap de facto inconsistent: The element is still physically stored in the internal data array, but cannot be accessed via regular access paths, such asget(key), can no longer be found. To demonstrate this effect reproducibly, consider the following code example:

javaCopy

importjava.util.*;classPerson{    Stringname;    youhand;    Person(Stringname,intid){        this.name=name;        this.id=id;    }    @Override    publicbooleanequals(Objecto){        returnoinstanceofPersonp&&Objects.equals(name,p.name)&&id==p.id;    }    @Override    publicinthashCode(){        returnObjects.hash(name,id);    }}publicclassMutableHashDemo{    publicstaticvoidmain(String[]args){        Personp=newPerson("Alice",42);        Map<Person,String>map=newHashMap<>();map.put(p,"Value");System.out.println("Before change:");        System.out.println("map.get(p): "+map.get(p));// Mutation of the key        p.name="Bob";System.out.println("After change:");        System.out.println("map.get(p): "+map.get(p));        System.out.println("Enthält key via entrySet: "+map.entrySet().stream()            .anyMatch(e->e.getKey().equals(p)));    }}

The demo source code is available on github athttps://github.com/svenruppert/Blog—Core-Java—Mutable-HashMap-Keys-in-Java/blob/main/src/test/java/junit/com/svenruppert/MutableHashCodeDemoTest.java

This example demonstrates thatmap.get(p) returns null after the change, although theentrySet() still contains the entry. The reasonhashCode() returns a different value after the mutation so that the original bucket is no longer addressed.equals() alone does not help in this case, since the lookup fails due to the wrong index.

Security-critical consequences: When loss of consistency becomes an attack surface

While the loss of referentiality in a HashMap may appear at first glance to be merely a technical issue, closer inspection reveals security-relevant implications, particularly in systems that utilise state caching, authentication, or access control. If an object serves as a key in security-critical maps or sets – e.g., for detecting active sessions, auth tokens, or user permissions – and its hash code subsequently changes, a logical error condition arises. The system no longer “sees” the object, even though it still exists. This can lead to unintended access gaps or, worse still, unattended persistence of state.

A particularly dangerous scenario arises when mutable keys can be influenced from outside. One conceivable scenario is where an attacker can inject controlled values ​​into an object that then serves as a key. As soon as this key is changed—for example, through manipulated API usage or faulty deserialisation— it is removed from access control, even though it technically still exists. The result: logical access without valid authorisation.

In extreme cases, this can even lead to typical security-critical patterns, such as:

• Authorisation Bypass: An object is modified before the test, thecontains() fails, and access is granted incorrectly.

• Resource Lock Hijack: A lock object is created viaa HashSet orMap managed by the system, but it can no longer be removed after mutation**–** a deadlock or race condition threatens.

• Denial of service due to hash collisions : If a system stores (or mutates) many mutable objects with controlled hash codes, hash collisions can be deliberately triggered, and performance problems can be created.

Although a classic buffer overflow doesn’t occur directly in Java due to the memory safety of the JVM model, the structural effect of an inconsistent HashMap is similar to an overflow at the logical level: An access lands in the “wrong memory area” (bucket), and the object is present but functionally invisible. This is a key point for security auditors and architects: Loss of consistency in hash-based structures can not only lead to incorrect behaviour but also become a potential entry point for complex attacks.

Interactive Demo with Vaadin Flow

To make the described problem tangible, a minimalist demonstration application is recommended, e.g., with Vaadin Flow. The demo aims to allow the user to observe live how an object in aHashMap becomes “invisible” after a mutation.

The application consists of a simple Vaadin view with the following UI elements:

• Input fields for name and ID
• A button to insert an object into a HashMap
• A button to modify the name (and thus the hash code)
• A button to execute map.get()
• A button for iterating over entrySet()

The view constructor first creates the graphical user interface. Two input fields – one for the name, one for the ID – are used to interact with thePerson -Object.

javaCopy

privatefinalTextFieldnameField=newTextField("Name");privatefinalTextFieldidField=newTextField("ID");privatefinalTextAreaoutput=newTextArea("Output");privatefinalPersonmutableKey=newPerson("Alice",42);privatefinalMap<Person,String>map=newHashMap<>();

Several buttons allow you to perform specific operations on theMap. The “put(key, value) " button adds the current mutable Key-Object as key with the value “Saved " to the map. This uses exactly the object whose name and id were determined at the beginning – here “Alice” and 42.

javaCopy

ButtonputButton=newButton("put(key, value)",_->{ map.put(mutableKey,"Saved");});

About the button“Change name”, the name of the mutableKey object at runtime. This leads to a state in which the contents of the object—and thus its hash code—change after it is inserted into the map.

javaCopy

ButtonmutateButton=newButton("Change name",_->{ mutableKey.name=nameField.getValue();});

The“get(key)” Button demonstrates this effect: The methodmap.get(mutableKey) attempts to retrieve the value using the (modified) object as a key.

javaCopy

ButtongetButton=newButton("get(key)",and->{ Stringresult=map.get(mutableKey); output.setValue("Result of get(): "+result);});

To illustrate this effect and, at the same time, provide an alternative access, the button“search entrySet()” has been added. It manually iterates through all key-value pairs of the map using the Stream API and compares the keys viaequals() , regardless of the internal bucket structure. This means that an entry with the changed key object can still be found, providedequals() is correctly implemented and independent of the hashCode function.

javaCopy

ButtoniterateButton=newButton("search entrySet()",and->{ Stringresult=map.entrySet().stream()     .filter(entry->entry.getKey().equals(mutableKey))     .map(Map.Entry::getValue)     .findFirst()     .orElse("Not found via equals()"); output.setValue("entrySet(): "+result);});//Here is the complete source code of the view.@Route(value=PATH,layout=MainLayout.class)publicclassVersionOneView   extendsVerticalLayout   implementsHasLogger{ publicstaticfinalStringPATH="versionone"; privatefinalTextFieldnameField=newTextField("Name"); privatefinalTextFieldidField=newTextField("ID"); privatefinalTextAreaoutput=newTextArea("Output"); privatefinalPersonmutableKey=newPerson("Alice",42); privatefinalMap<Person,String>map=newHashMap<>(); publicVersionOneView(){ logger().info("Initializing VersionOneView"); nameField.setValue(mutableKey.getName()); idField.setValue(String.valueOf(mutableKey.getId())); output.setWidth("600px"); ButtonputButton=newButton("put(key, value)",_->{   logger().info("Putting value into map with key: {}",mutableKey);   map.put(mutableKey,"Saved");   output.setValue("Inserted: "+mutableKey); }); ButtonmutateButton=newButton("Change name",_->{   logger().info("Changing name from {} to {}",mutableKey.getName(),nameField.getValue());   mutableKey.setName(nameField.getValue());   output.setValue("Name changed to: "+mutableKey.getName()); }); ButtongetButton=newButton("get(key)",and->{   logger().info("Getting value for key: {}",mutableKey);   Stringresult=map.get(mutableKey);   logger().info("Get result: {}",result);   output.setValue("Result of get(): "+result); }); ButtoniterateButton=newButton("search entrySet()",and->{   logger().info("Searching through entrySet for key: {}",mutableKey);   Stringresult=map.entrySet().stream()       .filter(entry->entry.getKey().equals(mutableKey))       .map(Map.Entry::getValue)       .findFirst()       .orElse("Not found via equals()");   logger().info("EntrySet search result: {}",result);   output.setValue("entrySet(): "+result); }); add(nameField,idField,putButton,mutateButton,getButton,iterateButton,output); logger().info("VersionOneView initialization completed");} publicclassPerson{   Stringname;   intid;   publicPerson(Stringname,intid){     this.name=name;     this.id=id;   }//SNIP getter setter   @Override   publicbooleanequals(Objecto){     returntheinstanceofPersonp             &&Objects.equals(name,p.name)             &&id==p.id;   }   @Override   publicinthashCode(){     returnObjects.hash(name,id);   }   @Override   publicStringtoString(){     returnname+" ("+id+")";   } }

This concise yet powerful view enables any reader to directly observe the effects of changing keys.

The demo source code is available on github athttps://github.com/svenruppert/Blog—Core-Java—Mutable-HashMap-Keys-in-Java/blob/main/src/main/java/com/svenruppert/flow/views/version01/VersionOneView.java** **

Now, let’s modify the view slightly and display all entries from the EntrySet. The hash codes are compared, and the result is output.

javaCopy

ButtoniterateButton=newButton("search entrySet()",_->{ logger().info("Searching through entrySet for key: {}",mutableKey); StringBuilderresult=newStringBuilder(); map.forEach((key,value)->{   booleanisMatch=key.equals(mutableKey);   result.append(String.format("""                                   Key: %s, Value: %s, HashCode %s Match: %s""",                               key,                               value,                               key.hashCode(),                               isMatch?"And":"No"));   result.append("\n"); }); logger().info("EntrySet search result: {}",result); output.setValue("entrySet():\n"+(!result.isEmpty()?result.toString():"Map is empty"));});

Now you can clearly see how an instance now exists multiple times in the same hash map. If you think about it, you can think of some very unpleasant applications that could even be used to specifically attack a system. But more on that in another blog post.

HashSet

Since aHashSet internally is nothing other than aHashMap , where the keys are the actual set elements and the values ​​are a constant dummy (such asPRESENT = new Object()), all the problems described here apply in the same way. If an object is modified after being inserted into the set, and this change affects one of the attributes that is used in the calculation ofhashCode() , the object is searched in the wrong bucket. The methodcontains() returnsfalse, even though the object is stored in the set.remove() fails because the same logic as get() takes effect: theHashMap finds the bucket using the current hash code and does not recognise the key there.

This behaviour is particularly critical whenHashSet is used to temporarily store security-relevant information, for example, to manage already authenticated users, valid tokens, or temporary permissions. An unintentionally mutated key can lead to access being incorrectly denied or even security checks being bypassed—a classic case of a logical security vulnerability that is barely noticeable and difficult to debug.

In safety-critical modules, set elements should therefore always be immutable and defensively constructed. This means, in concrete terms, no changeable fields, no public modifiability, and – ideally – construction via arecord or builder with final attributes.

xmlCopy

Set<Person> people = new HashSet<>();people.add(p); // p.hashCode() ist Xp.name = "Malicious";System.out.println(people.contains(p)); // false

Strategies to avoid

The simplest and most effective strategy is to use only immutable objects as keys. Since Java 16,records are the idiomatically correct tool for this. They automatically generate equals() and hashCode() based on final fields.

record PersonKey(String name, int id) {}

Alternatively, if mutable state is unavoidable, the objectbefore A modification can remove the hash code from the map and then reinsert it with an updated hash code. This is a dangerous workaround that is rarely implemented correctly in practice.

Why are other map implementations not affected

The problem described here occurs specifically inHashMap and structures based on it, such asHashSet, because a calculated hash code determines the access path. Other map implementations in the JDK – such asTreeMap ,LinkedHashMap , orEnumMap - are not affected in this respect or only to a lesser extent.

TheTreeMap, for example, is not based on hash code-based indexing, but on a sorted tree structure (red-black tree). It uses either the natural order of the keys (via Comparable) or a provided Comparator. This means that access does not depend on the result ofhashCode() but from the stable comparability throughcompareTo() orcompare(K1, K2). A change to an attribute that is included in the comparison logic can also lead to inconsistent behaviour, for example,get() orremove(). However, the mechanism is more transparent and controllable because sorting is done using a clearly defined comparison function.

Also,LinkedHashMap , although internally aHashMap , is not immune to the mutable hashcode problem because it uses the same bucket logic. However, it also provides a deterministic iteration order, which can aid in debugging, but does not alter the underlying problem.

TheEnumMap. Finally, it is protected against this problem by design, as it is exclusively of enum types as keys. These are immutable by language definition and have stable, final hash codes. Thus, mutation is impossible, and the map remains robust against this class of errors.

You should therefore carefully consider which map implementation is used in each context, and whether a stable key state can be guaranteed or whether alternative ordering mechanisms should be used.

Conclusion: The price of convenience

In daily development, the problem is often overlooked because it only manifests itself in specific situations – for example, after multiple operations, in multi-threading, or integrations across API boundaries. However, the consequences are severe: lost entries, unexplainednull values , and inconsistent state.

Anyone who wants to useHashMaps efficiently and correctly should respect their internal rules, particularly ensuring that key objects do not change subsequently.

Demo source code to try it out yourself

The complete source code for the demo view with Vaadin Flow is publicly available on GitHub. The application can be run efficiently locally using’ mvn jetty: run’. The web application will then be available at the addresshttp://localhost:8080/.

GitHub:https://github.com/svenruppert/Blog---Core-Java---Mutable-HashMap-Keys-in-Java

Happy Coding

Sven

]]>JavaSecure Coding PracticesUncategorizedhttps://svenruppert.com/posts/creating-a-simple-file-upload-download-application-with-vaadin-flow/Tue, 20 May 2025 17:34:15 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/creating-a-simple-file-upload-download-application-with-vaadin-flow/Vaadin Flow is a robust framework for building modern web applications in Java, where all UI logic is implemented on the server side. In this blog post, we’ll make a simple file management application step by step that allows users to upload files, save them to the server, and download them again when needed. This is a great way to demonstrate how to build protection against CWE-22, CWE-377, and CWE-778 step by step.<![CDATA[

Vaadin Flow is a robust framework for building modern web applications in Java, where all UI logic is implemented on the server side. In this blog post, we’ll make a simple file management application step by step that allows users to upload files, save them to the server, and download them again when needed. This is a great way to demonstrate how to build protection against CWE-22, CWE-377, and CWE-778 step by step.

1. Basic project structure
2. Add file upload functionality.
3. Add download functionality
4. Best Practices uses Java NIO.
5. CWE-22: Path Traversal and Protection Measures
6. CWE-377: Insecure temporary files
7. CWE-778: Insufficient logging
8. Summary

In this example, we’re focusing exclusively on functionality rather than on graphic design. The latter has been intentionally kept very simple to focus on the technical aspects.

The source texts for this article can be found at:https://github.com/Java-Publications/Blog—Secure-Coding-Practices—CWE-022–377–778—A-practical-Demo** __**

Basic project structure

To begin, we will create a new Vaadin project. The easiest way to do this is via the project starter, which you can find athttps://start.vaadin.com/ or by using an existing Maven template. The file structure of our project essentially looks like this:

Screenshot

The file**MainView.java** will be the application’s central entry point. Here, we will implement the user interface and the logic for file uploads and downloads.

Add file upload functionality.

First, we will create a simple user interface that allows users to upload files. In**MainView.java**, our basic setup looks like this:

javaCopy

packagecom.svenruppert.filemanager;importcom.vaadin.flow.component.button.Button;importcom.vaadin.flow.component.notification.Notification;importcom.vaadin.flow.component.orderedlayout.VerticalLayout;importcom.vaadin.flow.component.upload.Upload;importcom.vaadin.flow.component.upload.receivers.MemoryBuffer;importcom.vaadin.flow.router.Route;importjava.io.File;importjava.io.FileOutputStream;importjava.io.IOException;importjava.io.InputStream;@RoutepublicclassMainViewextendsVerticalLayout{    publicMainView(){        MemoryBufferbuffer=newMemoryBuffer();        Uploadupload=newUpload(buffer);        upload.setMaxFiles(1);        upload.addSucceededListener(event->{            StringfileName=event.getFileName();            try(InputStreaminputStream=buffer.getInputStream()){                FiletargetFile=newFile("uploads/"+fileName);                targetFile.getParentFile().mkdirs();                try(FileOutputStreamoutputStream=newFileOutputStream(targetFile)){                    inputStream.transferTo(outputStream);                }Notification.show("File "+fileName+" uploaded successfully!");            }catch(IOExceptione){Notification.show("Error uploading file");            }        });        add(upload);    }}

In this code, we useMemoryBuffer to temporarily save the uploaded file and then write it to theuploads/ directory. If the target directory doesn’t already exist, it will be created automatically. Using**MemoryBuffer** allows easy and secure file management before it is written to the hard disk.

Add download functionality

To list the downloadable files, we’ll add a button that allows users to download any file from the directory. This improves the user experience and ensures users can access their uploaded files easily. Here we’ll extend the user interface:

javaCopy

importcom.vaadin.flow.component.grid.Grid;importcom.vaadin.flow.component.html.Anchor;importjava.io.File;publicclassMainViewextendsVerticalLayout{    publicMainView(){// Upload function as described above        MemoryBufferbuffer=newMemoryBuffer();        Uploadupload=newUpload(buffer);        upload.setMaxFiles(1);        upload.addSucceededListener(event->{            StringfileName=event.getFileName();            try(InputStreaminputStream=buffer.getInputStream()){                FiletargetFile=newFile("uploads/"+fileName);                targetFile.getParentFile().mkdirs();                try(FileOutputStreamoutputStream=newFileOutputStream(targetFile)){                    inputStream.transferTo(outputStream);                }Notification.show("File "+fileName+" uploaded successfully!");                updateFileList();            }catch(IOExceptione){Notification.show("Error uploading file");            }        });        add(upload);        updateFileList();    }    privatevoidupdateFileList(){        removeAll();        Filefolder=newFile("uploads");        File[]listOfFiles=folder.listFiles();        if(listOfFiles!=null){            for(Filefile:listOfFiles){                if(file.isFile()){                    AnchordownloadLink=newAnchor("/uploads/"+file.getName(),file.getName());                    downloadLink.getElement().setAttribute("download",true);                    add(downloadLink);                }            }        }    }}

Using the method**updateFileList()** displays the files stored in theuploads/ directory as a list, and creates ananchor element for each file, which serves as a download link. This makes the interface more intuitive and allows users to manage uploaded files easily.

Best Practices uses Java NIO.

We can use Java NIO (New Input/Output) instead of traditional IO streams to improve efficiency and security when handling files. Java NIO provides non-blocking IO operations, enabling better performance and scalability. It also supports more flexible and secure file system operations.

We adapt our code to use Java NIO classes like**Files** and**Path** to save the uploaded files. Here’s an improved version of the upload code that uses Java NIO:

javaCopy

importjava.nio.file.Files;importjava.nio.file.Path;importjava.nio.file.Paths;importjava.nio.file.StandardCopyOption;publicclassMainViewextendsVerticalLayout{    publicMainView(){        MemoryBufferbuffer=newMemoryBuffer();        Uploadupload=newUpload(buffer);        upload.setMaxFiles(1);        upload.addSucceededListener(event->{            StringfileName=event.getFileName();            PathtargetPath=Paths.get("uploads").resolve(fileName);            try(InputStreaminputStream=buffer.getInputStream()){                Files.createDirectories(targetPath.getParent());                Files.copy(inputStream,targetPath,StandardCopyOption.REPLACE_EXISTING);Notification.show("File "+fileName+" uploaded successfully!");                updateFileList();            }catch(IOExceptione){Notification.show("Error uploading file");            }        });        add(upload);        updateFileList();    }}

This version uses theFiles andPaths classes to create directories and store files. This improves code readability and maintainability and leverages the advantages of the NIO classes, such as better exception handling and more flexible path operations. Using Java NIO makes file operations more efficient and secure, especially when working with multiple threads concurrently.

CWE-22: Path Traversal and Protection Measures

CWE-22, also known as “Path Traversal,” is a vulnerability that occurs when users can access unauthorised files and directories in the file system via insecure path names. This is typically done by users inserting special character strings such as../ into filenames to extend beyond the boundaries of the permitted directory. If this is not controlled correctly, an attacker could gain access to critical system files, potentially leading to a serious security compromise.

In our file management application, path traversal is risky if the filename is used directly and without verification to save or retrieve a file in the file system. Attackers could attempt to manipulate path information and overwrite or read files outside the intended directory.

To avoid vulnerabilities like CWE-22 (Path Traversal) in your application, you should be especially careful with user-supplied filenames. Attackers could attempt to access files outside their intended location using manipulated path names—for example, by entering something like../../etc/passwd. Therefore, checking and cleaning all paths and file names is essential before using them. Here are a few best practices you should implement:

• Path cleanup: Use methods like Path.normalize()from the Java NIO package to clean paths. This automatically removes dangerous constructs such as double slashes or relative elements (..) that attackers could use to traverse directories.

• Directory restriction: Make sure that the path you use to save files is within a secure, predefined home directory – e.g. uploads/ You can verify this by comparing the cleaned destination path with the expected base directory. You should allow the upload only if the destination path is truly within the allowed range.

• File name validation: It’s not enough to check the path alone. The filename itself can also contain dangerous characters. It’s best to allow only simple, non-critical characters, such as letters, numbers, hyphens, and periods. Using a regular expression like[a-za-Z0-9._—], you can specifically remove or replace everything else.

Considering these points makes it significantly more difficult for attackers to control paths and protect your server structure and users’ sensitive data.

Here is a customised version of our application that implements protections against CWE-22:

javaCopy

importjava.nio.file.Files;importjava.nio.file.Path;importjava.nio.file.Paths;importjava.nio.file.StandardCopyOption;publicclassMainViewextendsVerticalLayout{    publicMainView(){        MemoryBufferbuffer=newMemoryBuffer();        Uploadupload=newUpload(buffer);        upload.setMaxFiles(1);        upload.addSucceededListener(event->{            StringfileName=sanitizeFileName(event.getFileName());            PathtargetPath=Paths.get("uploads").resolve(fileName).normalize();// Prevent the path from being outside the upload directory            if(!targetPath.startsWith(Paths.get("uploads").toAbsolutePath())){Notification.show("Invalid file path! Upload aborted.");                return;            }            try(InputStreaminputStream=buffer.getInputStream()){                Files.createDirectories(targetPath.getParent());                Files.copy(inputStream,targetPath,StandardCopyOption.REPLACE_EXISTING);Notification.show("File "+fileName+" uploaded successfully!");                updateFileList();            }catch(IOExceptione){Notification.show("Error uploading file");            }        });        add(upload);        updateFileList();    }    privateStringsanitizeFileName(StringfileName){        returnfileName.replaceAll("[^a-zA-Z0-9._-]","_");    }}

In this updated version of the application, we’ve added asanitizeFileName() method to ensure the filename doesn’t contain any dangerous characters. We also normalise the path withPath.normalize() and verify that the final path is within the desireduploads directory. If the path is outside this directory, the upload is aborted and an appropriate error message is displayed.

These changes ensure that attackers cannot gain unauthorised access to the file system by manipulating file names. This keeps the application secure and protects both the server and the data stored on it from misuse.

CWE-377: Insecure temporary files

CWE-377, also known as “Insecure Temporary File,” describes a security vulnerability that occurs when you create temporary files in a way that can be exploited by attackers. Such files are often used to cache content, either during processing or for temporary storage in general. However, if you create them insecurely, attackers could access, tamper with, or even overwrite them.

A typical attack scenario: An attacker creates a symbolic link (symlink) pointing to a critical system file – your application then unknowingly writes data there. Or they manipulate the file while it’s still being processed, thereby introducing unwanted content or malicious code into the system. The consequences can range from data loss and integrity violations to a complete system compromise.

Temporary files are also created in your file management application—for example, when processing uploads. Therefore, you must create these files securely. Here are a few best practices to effectively avoid CWE-377:

• Safe methods for creating temporary files: In Java, use the Files.createTempFile() method. It automatically creates a temporary file with a unique, random name, reducing the risk of attackers accessing it.

• Restrict access rights: Only your process can access the temporary file. Set the file permissions so no other users or services have access—this is especially important in shared environments.

• Use unpredictable file names: Avoid using fixed or easily guessed names for temporary files. Otherwise, attackers could create a file with the same name beforehand or overwrite existing ones.

Considering these points, you can securely handle temporary files and protect your application from an often underestimated attack vector.

javaCopy

importjava.nio.file.Files;importjava.nio.file.Path;importjava.nio.file.Paths;importjava.nio.file.StandardCopyOption;publicclassMainViewextendsVerticalLayout{    publicMainView(){        MemoryBufferbuffer=newMemoryBuffer();        Uploadupload=newUpload(buffer);        upload.setMaxFiles(1);        upload.addSucceededListener(event->{            StringfileName=sanitizeFileName(event.getFileName());            PathtargetPath=Paths.get("uploads").resolve(fileName).normalize();// Prevent the path from being outside the upload directory            if(!targetPath.startsWith(Paths.get("uploads").toAbsolutePath())){Notification.show("Invalid file path! Upload aborted.");                return;            }            try(InputStreaminputStream=buffer.getInputStream()){// Create a secure temporary file                PathtempFile=Files.createTempFile("upload_","tmp");                Files.copy(inputStream,tempFile,StandardCopyOption.REPLACE_EXISTING);// Move the temporary file to the target directory                Files.createDirectories(targetPath.getParent());                Files.move(tempFile,targetPath,StandardCopyOption.REPLACE_EXISTING);Notification.show("File "+fileName+" uploaded successfully!");                updateFileList();            }catch(IOExceptione){Notification.show("Error uploading file");            }        });        add(upload);        updateFileList();    }    privateStringsanitizeFileName(StringfileName){        returnfileName.replaceAll("[^a-zA-Z0-9._-]","_");    }}

In this revised version, you create a secure temporary file with Files.createTempFile(). You use this to temporarily store the uploaded content before moving it to the final destination directory, ensuring that third parties cannot tamper with the temporary file.

Only after the content has been successfully saved do you move the file to the correct location in the file system. This way, you can upload the files in a controlled and secure manner and minimise the risk of temporary files becoming a gateway for attackers.

CWE-778: Insufficient logging

CWE-778, also known as “Insufficient Logging,” describes a security vulnerability that occurs when your application doesn’t log enough to detect and track security-relevant events. If you don’t maintain detailed logging, attempted attacks, unauthorised access, or system errors may go unnoticed for a long time, or you may not have enough information to respond appropriately in an emergency.

Especially in safety-critical applications, you should document all critical actions and errors to understand what happened later clearly. This is the only way to allow yourself or your team to respond quickly to incidents and learn from them.

If you log too little or nothing at all, you may miss signs of attacks, such as suspicious file names (path traversal), repeated failed uploads, or unauthorised access. To prevent this, you should keep a few basic things in mind:

• Log security-relevant events: Log all critical actions, such as file uploads, file accesses, path checks, or rejected requests, along with timestamps and context.

• Record errors and exceptions: Make sure that you display a message for all errors and exceptions and record the exact cause in the log. This allows you to search for specific sources of errors later.

• Use a central logging solution: Use proven logging frameworks like SLF4J in combination with Logback or Log4j2. This ensures that all log messages are recorded consistently, structured, and configurable for your environment.

Below is a customised version of the application, which allows you to insert log messages at security-critical points. This gives you essential insights during operation and will enable you to react quickly to problems.

javaCopy

importjava.nio.file.Files;importjava.nio.file.Path;importjava.nio.file.Paths;importjava.nio.file.StandardCopyOption;importorg.slf4j.Logger;importorg.slf4j.LoggerFactory;publicclassMainViewextendsVerticalLayout{    privatestaticfinalLoggerlogger=LoggerFactory.getLogger(MainView.class);    publicMainView(){        MemoryBufferbuffer=newMemoryBuffer();        Uploadupload=newUpload(buffer);        upload.setMaxFiles(1);        upload.addSucceededListener(event->{            StringfileName=sanitizeFileName(event.getFileName());            PathtargetPath=Paths.get("uploads").resolve(fileName).normalize();// Prevent the path from being outside the upload directory            if(!targetPath.startsWith(Paths.get("uploads").toAbsolutePath())){Notification.show("Invalid file path! Upload aborted.");logger.warn("Invalid file path attempt: {}",targetPath);                return;            }            try(InputStreaminputStream=buffer.getInputStream()){// Create a secure temporary file                PathtempFile=Files.createTempFile("upload_","tmp");                Files.copy(inputStream,tempFile,StandardCopyOption.REPLACE_EXISTING);// Move the temporary file to the target directory                Files.createDirectories(targetPath.getParent());                Files.move(tempFile,targetPath,StandardCopyOption.REPLACE_EXISTING);Notification.show("File "+fileName+" uploaded successfully!");logger.info("File {} successfully uploaded to {}",fileName,targetPath);                updateFileList();            }catch(IOExceptione){Notification.show("Error uploading file");logger.error("Error uploading file {}: {}",fileName,e.getMessage(),e);            }        });        add(upload);        updateFileList();    }    privatevoidupdateFileList(){        removeAll();        Filefolder=newFile("uploads");        File[]listOfFiles=folder.listFiles();        if(listOfFiles!=null){            for(Filefile:listOfFiles){                if(file.isFile()){                   AnchordownloadLink=newAnchor("/uploads/"+file.getName(),file.getName());                    downloadLink.getElement().setAttribute("download",true);                    add(downloadLink);logger.info("File {} added to download list",file.getName());                }            }        }else{logger.warn("Directory 'uploads' could not be read or is empty.");        }    }    privateStringsanitizeFileName(StringfileName){        returnfileName.replaceAll("[^a-zA-Z0-9._-]","_");    }}

The revised implementation uses the SLF4J (Simple Logging Facade for Java) logging API in combination with a specific implementation such as Logback. This separation of API and logging engine offers flexibility in choosing the underlying technology and facilitates integration into different runtime environments. The goal is to systematically record security-relevant events to ensure both transparency and traceability of security-critical actions during operation.

In the context of security-conscious web applications—especially those that work with user-generated content such as file uploads—comprehensive and structured logging is essential. You should explicitly record the following event types:

• Path traversal tests (CWE-22): If a user attempts to access paths outside the intended memory area through manipulated input, you should report this with a WARN or higher log level. The logged information should include both the compromised path and the associated session or IP address to enable later correlation with other events.

• Successful file uploads: Every completed upload process should be documented in the log, including the cleaned-up file name, the target path in the file system, and optional context information such as the user ID or upload time. This allows for complete traceability and also serves as an audit trail.

• Upload error: If an exception occurs while saving the file (e.g., due to file system errors, access violations, or corrupted streams), you should log the full stack trace and all relevant metadata (file name, user context, time). This is essential for efficient error diagnosis and allows you to distinguish potential attack attempts from legitimate error cases.

• Dynamic update of the file view: Logging can also be helpful for internal system actions, such as refreshing the list of available files to track when a file becomes visible or whether problems occurred during processing (e.g., access errors in locked directories).

Implementing these measures consistently allows you to detect and evaluate security-relevant incidents during ongoing operations promptly. Furthermore, a transparent logging strategy ensures that administrators and incident response teams have a solid database to rely on during an investigation. This strengthens your application’s ability to respond to attacks and fulfils key requirements for traceability, auditing, and compliance in security-critical IT systems.

Summary

We developed a functional file management application in Vaadin Flow in just a few steps. Users can upload files securely stored in the server directory and download them again. We used Java NIO to make file handling more efficient and secure. Additionally, we implemented safeguards against CWE-22 (Path Traversal), CWE-377 (Insecure Temporary File), and CWE-778 (Insufficient Logging) to ensure that attackers cannot gain unauthorised access to the file system, that temporary files are created securely, and that security-relevant events are comprehensively logged. This example demonstrates how easy it is to create a powerful user interface and implement server-side logic in Java with Vaadin Flow.

The next steps could be to extend the application, for example, by adding user permissions, customising the file overview, or integrating a search function for uploaded files. Vaadin offers numerous visual components to further improve the usability of your applications, but it is up to the developer to secure these features. Furthermore, we could extend the application with features such as drag-and-drop for file uploads, user role management, or a connection to a database for indexing files and storing metadata. However, with each additional feature, there are also additional attack vectors. It’s always important to find the right balance.

By enhancing security measures such as implementing robust file validation, sufficient logging mechanisms, and using Java NIO, we ensure that the application remains secure and efficient for both the developer and end users.

We’ll explore different aspects in the following parts, so it remains exciting.

Happy Coding

Sven

]]>JavaSecure Coding PracticesUncategorizedVaadinhttps://svenruppert.com/posts/open-hearted-bytecode-java-instrumentation-api/Fri, 11 Apr 2025 17:06:44 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/open-hearted-bytecode-java-instrumentation-api/What is the Java Instrumentation API? The Java Instrumentation API is part of the java.lang.instrument package and allows you to change or analyse class bytecode at runtime. It is particularly intended for the development of profilers, agents, monitoring tools, or even dynamic security mechanisms that need to intervene deeply in a Java application’s behaviour without changing the source code itself.<![CDATA[

What is the Java Instrumentation API?

The Java Instrumentation API is part of the java.lang.instrument package and allows you to change or analyse class bytecode at runtime. It is particularly intended for the development of profilers, agents, monitoring tools, or even dynamic security mechanisms that need to intervene deeply in a Java application’s behaviour without changing the source code itself.

1. What is the Java Instrumentation API?
2. What are the advantages and disadvantages?
   1. Advantages
   2. Disadvantages
3. What are the performance implications?
   1. Influence during class loading
   2. Impact at runtime
   3. Memory and garbage collector effects
4. What security implications arise?
   1. Security implications in detail
   2. Relevant CWEs (Common Weakness Enumerations)
5. A practical example
   1. 1. REST service without external dependencies
   2. 2. Java Agent for instrumentation
   3. How does the whole thing start?
   4. Why do you need one? MANIFEST.MF?
   5. What does a valid one look like? MANIFEST.MF out of?
   6. Where and how is the file created?
   7. How does this work at runtime?
   8. What and why is this happening?

At the heart of this API is the concept ofJava Agents. An agent is a special component that is activated when the JVM starts or at runtime (via the so-calledAttach API) can be loaded. The agent then uses an Instrumentation-Interface passed by the JVM launcher. This interface allows the agent to inspect already loaded classes, transform new classes or influence the behavior of the JVM - for example, by inserting hooks, measuring call times or injecting security checks.

The basic task is to transform the bytecode during the classloading process without violating the semantics of the Java language. Technically, this is done by a so-called ClassFileTransformer, which is registered and can manipulate the bytecode every time a class is loaded before it is interpreted or compiled by the JVM.

A typical entry point for a static agent is a class with a premain(String agentArgs, Instrumentation inst)-Method. With dynamic agents, this comes instead agentmain(String agentArgs, Instrumentation inst)method to use. Both variants allow transformers to be registered and thus influence the behavior of the application below the language level.

Using the Instrumentation API requires a deep understanding of the JVM architecture and Java class loading mechanism. At the same time, it also opens up unique possibilities for runtime analysis and modification that would not be achievable using normal means in Java, and in a way that can be elegantly integrated into existing applications without having to recompile them.

What are the advantages and disadvantages?

Advantages

A key advantage lies in the ability tomanipulatebytecodes transparently : You can change the behavior of existing classes without touching their source code. This is particularly relevant in safety-critical or highly dynamic environments, such as when inserting logging, tracing or metrics code. The API also enables the so-calledClass Retransformation , which means that already loaded classes can be subsequently changed under certain conditions. This feature is used intensively in conjunction with tools such as JRebel or modern observability solutions such as OpenTelemetry.

Another advantage is integration without source code changes. An agent can be started using the Java process or added at runtime without the target application even knowing it is being instrumented. This is particularly essential for debugging, monitoring or security solutions.

In addition, the Instrumentation API has beenPart of the JDK since Java 5 and requires no external dependencies in security-conscious environments—for example, ​​critical infrastructure or the government environment—this can be a significant argument for but also against its use.

Disadvantages

The power of the Instrumentation API inevitably brings significant disadvantages. First and foremost is theComplexity of bytecode manipulation. Although there are libraries such as ASM or Byte Buddy that make working with bytecode easier, getting started remains deep in the JVM’s engine room and requires a sound understanding of the Java class structure, class loading, and bytecode specification.

A second risk concernsStability and predictability : Interference with the bytecode can lead to subtle errors, for example, when method signatures are changed, security checks are bypassed, or synchronisation blocks are manipulated. Such errors are difficult to test and can only occur under certain runtime conditions, drastically increasing the debugging effort.

Those tooCompatibility across different JVM versions is a problem. Not every JVM behaves identically when it comes to class loading or transformation. Particularly with GraalVM or in AOT environments, certain features of the Instrumentation API may not work or only work to a limited extent. This particularly applies to dynamic reloading of classes or access to low-level JVM internals.

After all, performance overhead should not be neglected. While simply registering an agent is usually very efficient, when transformers perform complex operations on every classload or inject metrics, this can result in measurable startup or runtime delays, especially in I/O-intensive server applications.

The Java Instrumentation API is a double-edged sword: It offers a unique opportunity to intervene in the behavior of a Java application deeply, but at the price of increased complexity, potential instability, and difficult maintainability. Its use should, therefore, be well justified, targeted, and accompanied by appropriate testing and logging infrastructure. However, it remains an indispensable tool in safety-critical or highly dynamic contexts where application code modification is impossible.

What are the performance implications?

The performance implications of the Java Instrumentation API are complex and depend heavily on the respective use case, the complexity of the transformations carried out, and the type of target application. The Instrumentation API influences performance during three phases: during class loading, at runtime of the instrumented classes, and in connection with memory or security-relevant operations.

Influence during class loading

The most significant immediate impact occurs when the class loader loads a class. As soon as a ClassFileTransformer has been registered, it is called every time a new class is loaded. At this moment, the JVM passes the class’s original bytecode to the Transformer, which can modify it or return it unchanged. The duration of this transformation directly affects the application’s start time or the dynamic reloading of classes.

Complex transformations, especially those performed with ASM or similar bytecode libraries, can quickly generate measurable overhead, especially in framework-based applications with thousands of classes. If additional logging, tracing or code injections are carried out in each method, the loading time increases exponentially because each method block has to be processed individually.

Impact at runtime

Additional overhead may also arise at runtime, depending on how the bytecode was changed. This is particularly true if the Transformer inserts additional control flow, such as logging instructions, time measurement, additional validations or security-related checks. For example, such insertions can extend the execution of each method by several nanoseconds to milliseconds, which can lead to massive performance losses in methods called at high frequencies (e.g. in I/O loops or business logic).

A typical example is the so-called method entry/exit tracing, where logging code is injected before and after each method. Even if this logging code itself does not actively write, but only sets passive markers, a memory and CPU load can still add up under high load.

In addition, the instrumentation can alsoJIT optimizations affect the JVM , especially when unpredictable control flows are inserted or methods are artificially inflated. This can lead to certain hotspots no longer being converted into natively optimised code, with corresponding consequences for execution time.

Memory and garbage collector effects

Introducing additional object state during instrumentation—for example, through static caches, trace information, or associated metadata—can increase heap usage. This becomes particularly critical if weak or soft references are not managed correctly or if new classes with slightly different structures are continually created, which, in the worst case, can lead to an OutOfMemoryError in the class memory.

The Java Instrumentation API, when used efficiently and purposefully, can be used with minimal impact on performance. However, this requires that the registered transformers are kept as lean as possible, unnecessary transformations are avoided, and classes are filtered in a targeted manner. At the same time, one should always keep in mind that even small changes in the bytecode can have far-reaching effects on the JVM’s optimization strategies and memory behavior. Therefore, careful benchmarking and targeted monitoring are essential if you want to use the API in production systems.

What security implications arise?

Because it can manipulate classes at runtime, remove security checks, or inject new bytecode, it opens up an attack surface far beyond what typical application code can do. It therefore represents a potential gateway for privilege escalation, code injection and persistence mechanisms, such as those observed with Advanced Persistent Threats (APT) or malicious agents.

Security implications in detail

First of all, it should be noted that the Instrumentation API can deliberately operate past the JVM sandbox. Once an agent is active - be it when it is started via the -javaagent argument or dynamically via the Attach API - it has almost complete control over all loaded classes in the context of the running JVM. It can remove security checks, alter stack traces, redirect control flow, or extract sensitive information from memory. Also, the transformation of classes from security-relevant packages such as java.lang.reflect or javax.crypto is possible if no SecurityManager restrictions are active (which, however, have been deprecated since Java 17 and are completely eliminated from Java 21).

The Attach API is particularly critical because it allows arbitrary Java processes on the local system to be provided with an agent at runtime—as long as the same user context is present. This means that an attacker who gains access to a shell account could compromise any of that user’s Java processes without that process having to be prepared for this. This scenario corresponds to a form ofruntime privilege escalation at the process level.

Relevant CWEs (Common Weakness Enumerations)

Concerning the Java Instrumentation API, several CWE categories can be named that are relevant in the context of JVM and agent manipulation:

**CWE-94: Improper Control of Generation of Code (‘Code Injection’) **Suppose an agent is used to generate or load new bytecode from external sources at runtime (e.g., network data, configuration files, or user input). In that case, this is a classic code injection, which can lead to a full takeover of the JVM.

**CWE-250: Execution with Unnecessary Privileges **By default, agents operate with the full rights of the Java process. Malicious agents can exploit unnecessary privileges if appropriate access protection is not implemented (e.g., through user separation or restrictive process isolation).

**CWE-269: Improper Privilege Management **Improper use of an agent to transform security-critical classes (e.g. ClassLoader, AccessController, Cryptographic Provider) can lead to a breach of the Java security model.

**CWE-272: Least Privilege Violation **If applications unnecessarily activate the Attach API or even provide their agent interfaces, this violates the principle of minimum rights allocation.

**CWE-749: Exposed Dangerous Method or Function **An improperly configured application, for example, making the Attach API available via an HTTP endpoint (e.g. in self-built monitoring solutions) opens dangerous functions to external attackers.

**CWE-798: Use of Hard-coded Credentials **In some dynamic agents, the transformation of security-relevant classes uses internal access or passwords that remain in the bytecode. These are easily extracted and can lead to the compromise of other systems.

From a security perspective, the Java Instrumentation API can be classified as a privileged interface, which, if handled incorrectly, leads to significant opportunities for attacks. Your use should therefore always be carried out withprocedural and technical controls flanked - for example, by isolating agents in dedicated containers, by restrictive file system and user rights, and by consistently deactivating the Attach API in production environments using the JVM option -Dcom.sun.management.jmxremote=false or -XX:+DisableAttachMechanism.

Careful use of instrumentation is essential, particularly in security-critical scenarios, such as processing personal data or in distributed microservices architectures. Using this API without a sound security concept can quickly become a high-risk attack vector.

A practical example

A simple but well-founded example of using the Java Instrumentation API in the context of a REST service can be implemented using onboard tools in the JDK, particularly using the com.sun.net.httpserver.HttpServer for server functionality. The goal is to write a Java agent that transforms all REST endpoints during class loading so that they automatically insert logging statements at the start of each method without changing the endpoints’ original code.

The application,, therefore, consists of two parts:

1. The REST service based on HttpServer
2. A Java agent that works via Instrumentation, a ClassFileTransformer registered

1. REST service without external dependencies

The aim of this service is to provide simple license management based on GET and POST requests:

javaCopy

importcom.sun.net.httpserver.HttpServer;importcom.sun.net.httpserver.HttpHandler;importcom.sun.net.httpserver.HttpExchange;importjava.io.IOException;importjava.io.OutputStream;importjava.net.InetSocketAddress;publicclassLicenseRestServer{publicstaticvoidmain(String[]args)throwsIOException{HttpServerserver=HttpServer.create(newInetSocketAddress(8080),0);server.createContext("/license",newLicenseHandler());server.setExecutor(null);server.start();System.out.println("Server started on port 8080");}staticclassLicenseHandlerimplementsHttpHandler{@Overridepublicvoidhandle(HttpExchangeexchange)throwsIOException{if("GET".equals(exchange.getRequestMethod())){Stringresponse="Current license: DEMO-1234-5678";exchange.sendResponseHeaders(200,response.getBytes().length);try(OutputStreamos=exchange.getResponseBody()){os.write(response.getBytes());}}else{exchange.sendResponseHeaders(405,-1);// Method Not Allowed}}}}

This application starts a local REST server and responds GET /license with a static license key.

2. Java Agent for instrumentation

Now, a static Java agent is implemented that runs when the JVM starts -javaagent can be integrated. The goal is to have the class at runtime LicenseHandler, to transform and to precede each method with simple logging.

javaCopy

importjava.lang.instrument.ClassFileTransformer;importjava.lang.instrument.Instrumentation;importjava.security.ProtectionDomain;publicclassSimpleLoggingAgent{publicstaticvoidpremain(StringagentArgs,Instrumentationinst){System.out.println("[Agent] Initializing agent with arguments: "+agentArgs);inst.addTransformer(newLoggingTransformer());}staticclassLoggingTransformerimplementsClassFileTransformer{@Overridepublicbyte[]transform(Modulemodule,ClassLoaderloader,StringclassName,Class<?>classBeingRedefined,ProtectionDomainprotectionDomain,byte[]classfileBuffer){if(className!=null&&className.contains("LicenseRestServer$LicenseHandler")){System.out.println("[Agent] Transforming class: "+className);// Here you could modify real bytecode (e.g. with ASM)// For the purposes of this example, we return the original bytecode.}returnclassfileBuffer;}}}

In this minimal form, the transformer does not make any real bytecode changes, but it shows the basic process: when loading the class LicenseHandler the transform method is called. An extended version could be done here using the ClassReader and ClassWriter out of org.objectweb.asm Inject targeted logging at the beginning of each method.

How does the whole thing start?

1. Compiling both classes into separate JARs:

javaCopy

*license-service.jarcontainstheapplication.*agent.jarcontainstheagentandaMANIFEST.MFwithPremain-Class:SimpleLoggingAgent.

2. Starting the application with the agent:

java -javaagent:agent.jar -jar license-service.jar

When starting, the JVM loads the agent, then registers the transformer. As soon as the LicenseHandlerclass is loaded, the transformer gets access to its bytecode.

TheManifest file plays a central role in the context of a Java agent because it explicitly tells the JVM when it startswhich class initializes the agent – i.e. where the method is premain() or agentmain() located. Without this information, the JVM cannot recognize the agent class and will ignore the agent at startup. This is the technical reason why the manifest file is mandatory if you want to build a Java agent.

Why do you need one? MANIFEST.MF?

If an agent over -javaagent:path/to/agent.jar is started, the JVM reads the JAR and searches in META-INF/MANIFEST.MF by one of the following attributes:

Premain-Class : Specifies the class that will be used at JVM startup (before main()) is executed.

Agent-Class : Specifies the class that belongs to. Amore dynamic Agent attachment is used via the Attach API.

Can-Redefine-Classes, Can-Retransform-Classes, etc. : Optional, to specify additional capabilities.

Without a correctly placed one Premain-Class attribute, the agent cannot be started. The JVM has no mechanism to guess or scan this automatically.

What does a valid one look like? MANIFEST.MF out of?

A minimal manifest file for a static Java agent might look like this:

yamlCopy

Manifest-Version:1.0Premain-Class:com.svenruppert.securecoding.agent.SimpleLoggingAgentCan-Redefine-Classes:true

Ensure that each line has a line break (\n) end and that no line break is forgotten. Spaces are also meaningful – there must be exactly one space between the attribute and the value.

Where and how is the file created?

You can create the MANIFEST.MF file in several ways. In this example, I’m using Maven to declare the necessary parameters and attributes.

Example for pom.xml :

xmlCopy

<build><plugins><plugin><artifactId>maven-jar-plugin</artifactId><configuration><archive><manifestEntries><Premain-Class>com.svenruppert.securecoding.agent.SimpleLoggingAgent</Premain-Class><Can-Redefine-Classes>true</Can-Redefine-Classes></manifestEntries></archive></configuration></plugin></plugins></build>

How does this work at runtime?

If you start the JVM with the agent:

java -javaagent:simple-agent.jar -jar my-rest-service.jar

The JVM does the following:

• Read the JAR simple-agent.jar.
• Opens META-INF/MANIFEST.MF.
• Finds Premain-Class: SimpleLoggingAgent.
• Loads this class.
• Leads premain(String args, Instrumentation inst) out, still before the main()method of the actual program is started.

This gives the agent the chance to influence the behavior of the program before the first line of business logic is executed. This is precisely where his power lies – and the risk. The manifest file is the link between the JVM startup mechanism and the agent code. It is essential for Java agents because it tells the JVMwhich code actually contains agent logic. Their correct creation is a prerequisite for any form of static instrumentation. If you set it incorrectly or not at all, it will have no effect when the agent is started - the JVM will then ignore the agent without comment.

What and why is this happening?

This architecture shows the strength of the Instrumentation API: without changing a line of application code, functionalities such as logging, tracing, or metrics can be injected at runtime. This happens early in the JVM’s life cycle (before the actual program starts), which means that all future classes can be affected if specifically addressed.

Even without additional libraries, this setup demonstrates the core mechanism of the Instrumentation API: runtime manipulation of classes in a production-level environment. It shows how profound and dangerous this mechanism can be if there are no appropriate protection mechanisms at the JVM or operating system level.

Happy Coding

Sven

]]>Javahttps://svenruppert.com/posts/synchronous-in-chaos-how-parallel-collectors-bring-order-to-java-streams/Tue, 08 Apr 2025 20:25:45 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/synchronous-in-chaos-how-parallel-collectors-bring-order-to-java-streams/Sometimes it’s not enough for something to work - it has to work under load. In modern applications that process large amounts of data, the Streams API in Java provides developers with an elegant, declarative tool to transform, filter, and aggregate data in pipelines. ​​Describing complex data operations with just a few lines is seductive and realistic. But what happens when these operations encounter millions of entries? When should execution be done in multiple threads in parallel to save time and effectively use multi-core systems?<![CDATA[

Sometimes it’s not enough for something to work - it has to work under load. In modern applications that process large amounts of data, the Streams API in Java provides developers with an elegant, declarative tool to transform, filter, and aggregate data in pipelines. ​​Describing complex data operations with just a few lines is seductive and realistic. But what happens when these operations encounter millions of entries? When should execution be done in multiple threads in parallel to save time and effectively use multi-core systems?

1. Basics: Collector and concurrency
2. Criteria for parallelizable collectors
3. Examples from the Java standard library
4. Own parallel collector implementations
5. Best practices for productive use
6. Application scenario: License analysis in the benchmark
   1. The data model
   2. The test setup
7. Comparison with alternative parallelism concepts
8. Conclusion and outlook

At this point, a concept that often receives too little attention is the collector. The element at the end of a stream pipeline determineswhat should happen with the processed data. And even though the API seems simple – collect(Collectors.toList()) – there is an architecture behind it that brings its challenges when executed in parallel.

This text is, therefore, not just about the syntax or mechanics of collectors but rather a deep understanding of the conditions under which they can be used correctly and efficiently in parallel. We look at standard JDK solutions, discuss our implementations, show typical errors, and ultimately ask ourselves: How parallel can a collector be without becoming dangerous?

Basics: Collector and concurrency

At first glance, Java’s Streams API suggests that collecting results—the so-called terminal aggregation—can easily be carried out in parallel. But behind the method collect(…) hides more than just syntactic convenience. It is a coordinated collaboration between a data stream and a so-called collector—an object that forms a whole from individual parts.

A collector essentially consists of four functional components: the supplier, which provides a new cache for each subprocess; the accumulator, which feeds elements into this cache; the combiner, which merges multiple caches into one; and finally, the finisher, which produces the result. While supplier and accumulator are also essential in sequential streams, the combiner only comes into action when several threads have collected independently of each other, i.e., one parallelStream().

Here lies the first fundamental difference between sequential and parallel processing: in a sequential stream, it is sufficient to accumulate step by step into a single memory. In the parallel variant, on the other hand, several buffers are created that are isolated from one another, the contents of which must later be merged into a final result without conflict. This merging happens through the combiner, and it is precisely at this point that it is decided whether a collector is suitable for parallel use or not.

This suitability depends on several properties: The operations must be associative, i.e., deliver the same result regardless of the combination of the intermediate results. In addition, there must not be a shared state without synchronization. Last but not least, the individual steps must remain deterministic and free of side effects—otherwise, parallelisation quickly becomes a source of subtle errors.

Knowing these structural requirements is the first step toward consciously using parallel processing. Only if you understand how the collector and stream work together can you estimate when a performance gain is possible and when you will get unstable or simply wrong results instead.

Criteria for parallelizable collectors

Let’s imagine a stream running in parallel—perhaps over a large dataset divided into several segments. Each segment is now processed independently. What sounds trivial has profound implications: As soon as several threads collect at the same time, their intermediate results must not get in the way. The responsibility for correctness lies with the collector—or, more precisely, with its structural and functional design.

The first fundamental property isAssociativity. A combiner call must produce consistent results regardless of the order. combine (a, b) and combine (b, a) must produce equivalent results. This is necessary because the order of combination in a parallel context depends on the scheduler and is, therefore, unpredictable.

A second central point concernsAccess to memory structures. A potential hotspot for race conditions arises when a collector uses a shared, mutable state during accumulation, such as an out-of-sync list or map. The collector must either work exclusively with local, thread-isolated caches or rely on concurrent data structures such as b: ConcurrentHashMap, LongAdder, or explicitly synchronized wrappers.

In addition, it is alsodeterministic and an essential criterion: parallel execution must not lead to different results, neither in terms of content nor structure. Caution is advised, especially with non-deterministic structures like HashSet or HashMap, as the iteration order can vary, as with Collectors.joining() or Collectors.toMap() becomes problematic when the application relies on order.

These requirements—associativity, isolated state, and determinism—form the technical touchstone for parallel collectors. They are not optional but fundamental. Anyone who ignores them risks difficult-to-reproduce errors, incomplete results, or high-performance but semantically incorrect output.

Concurrency in Java Streams is powerful but not trivial. Only if a collector meets these criteria does it become one parallelStream() more than a placebo effect.

Examples from the Java standard library

An obvious way to make the abstract concept of parallel collectors tangible is through the collectors already included in the Java standard library. Many developers use Collectors.toList(), toSet(), or joining() almost every day—but rarely with knowledge of whether and how these collectors behave in a parallel context.

A simple example: The Collector Collectors.toList() uses an internal ArrayList. This is not thread-safe; therefore, the result when used in parallel is potentially inconsistent unless the buffers are isolated internally.

xmlCopy

public static<T>Collector<T,?,List<T>> toList() { return new CollectorImpl<>(ArrayList::new, List::add, (left, right) -> { left.addAll(right); return left; }, CH_ID);}

In fact, this collector still works correctly in parallel streams because the Streams API allocates each thread its accumulation area and merges only at the end via a combined merge process. The crucial point lies not in the data structure itself but in its controlled isolation.

It turns out to be less robust: Collectors.groupingBy(…). This variant is based on one HashMap and is not designed for concurrent access. This collector will be in one parallelStream(). Used without suitable protective measures, there is a risk of race conditions. The standard solution is Collectors.groupingByConcurrent(…), internal ConcurrentHashMap, designed for simultaneous access.

xmlCopy

public static<T,K>Collector<T,?,ConcurrentMap<K,List<T>>>groupingByConcurrent(Function<? super T, ? extends K> classifier) { return groupingByConcurrent(classifier, ConcurrentHashMap::new, toList());}

A look at the signature of this method already shows the intention:

xmlCopy

Map<Integer,List<String>> result = namen.parallelStream() .collect(Collectors.groupingByConcurrent(String::length));

In this example, strings are grouped by their length in a way that can be processed in parallel. Crucially, both the map implementation and the accumulation process are thread-safe.

Equally interesting Collectors.toConcurrentMap(…), which is explicitly intended to aggregate large sets of key-value pairs in parallel. The combination of key conflicts and the correct handling of merge functions is particularly interesting here.

These examples conclude that not every standard collector is suitable for parallel use per se. Just because a method from the Collectors-kit, this does not mean that it will work correctly in every execution configuration. The context decides - and with it the data structure used, the behavior of the combiner and the type of accumulation.

So, if you want to get more than one result from a stream—a correct and high-performance result—you should choose your collector just as carefully as you choose the filter criterion at the beginning of the pipeline.

Own parallel collector implementations

As powerful as the Java Standard Library’s prebuilt collectors are, they are sometimes not sufficient for specific needs. Especially when domain-specific aggregations, specialized data structures, or non-trivial reduction logic are required, it is worth considering the possibility of creating your own collector implementations.

A custom collector is usually created using the static method Collector.of(…) created. This method expects five parameters: one Supplier, which creates a new accumulator; a BiConsumer<A, T>, which inserts an element into the accumulator; a BinaryOperator for combining two accumulators; optionally one Function<A, R> to convert the result; and a Collector.Characteristics…-Array containing meta information like CONCURRENT or UNORDERED provides.

A simple but meaningful collector could, for example, collect strings in parallel ConcurrentLinkedQueue collect:

xmlCopy

Collector<String,?,Queue<String>> toConcurrentQueue() { return Collector.of( ConcurrentLinkedQueue::new, Queue::add, (left, right) -> { left.addAll(right); return left; }, Collector.Characteristics.CONCURRENT, Collector.Characteristics.UNORDERED );}

This collector is both CONCURRENT and UNORDERED, which means it can be written to by multiple threads simultaneously without guaranteeing the insertion order. What is important is that ConcurrentLinkedQueue acts as a thread-safe data structure, and the operation addAll is also incidentally uncritical.

However, more complex scenarios, such as the parallel determination of statistical key figures (minimum, maximum, average) across a data set, are also conceivable. In such cases, one can record serve as an accumulator structure that already encapsulates all the required partial states. The combiner then only has to consolidate these structures field by field.

Own collector implementations force you to deal intensively with the parallelizability of the data structures used and the combinability of the aggregation logic. This is not a disadvantage, but a valuable learning effect, because only those who understand what a collector does inside can use it consciously and safely.

Best practices for productive use

Anyone who wants to use collectors productively in a parallel context should consider some proven strategies - not as rigid rules, but as a framework for robust and efficient implementations.

A first principle is:Only parallelize if there is real benefit to be expected. Small amounts of data, trivial transformations or IO-bound processes usually do not benefit from parallelStream(). On the contrary: the overhead of thread management can even exceed the potential performance gain. Parallelization is only worthwhile if the amount of data to be processed is sufficiently large and the operations are CPU-intensive.

Second:Use only thread-safe or isolated data structures. This means that each thread uses its accumulator, which the Streams API supports internally, or explicitly concurrent data structures such as ConcurrentHashMap, ConcurrentLinkedQueue, or atomic wrappers can be used.

Third:Select collectors specifically. The standard library provides groupingByConcurrent, toConcurrentMap or mapping powerful tools specifically designed for parallel use. Anyone who also develops their solutions should pay particular attention to this combiner and the associativity of logic.

Fourth:Validate results , especially for new or complex pipelines. Parallel streams do not behave deterministically in execution, so tests in different utilization scenarios and under varying loads are necessary. This is especially true if collectors were developed or adapted in-house.

Last but not least:Measure instead of assume. Tools such as JMH (Java Microbenchmark Harness), Flight Recorder, or async-profiler help make realistic statements about performance advantages. Parallelization without metrics is like flying blind with a tailwind—it may be faster, but possibly in the wrong direction.

In summary, Parallel collectors are powerful tools, but they can only develop their potential if they are used consciously, well-founded, and with an eye on data structure, semantics, and execution context.

Application scenario: License analysis in the benchmark

In our example, we will now look at the following scenario. We will evaluate fictitious license information. These consist of several attributes that lend themselves to automated analysis: Is the license valid? Is this a trial version? Which country does the license come from? How long was it valid?

Instead of answering these questions individually and sequentially, the Java Streams API allows us to view the data as a stream and convert it into a consolidated analysis in a collector. But what happens if we analyse not 100, but 100,000 or 1,000,000 such license objects? A simple one is enough stream(), or is the parallel variant worth it? parallelStream()?

This is precisely where our experiment comes in: We want to measure the amount of data for which a parallel collector is noticeably worthwhile, both in terms of runtime and resource utilization. We use a specially developed collector that aggregates various metrics:

• Total number of valid licenses
• Share of test licenses
• Average duration in days
• Distribution of licenses by country
• The oldest and youngest licenses issued

To capture these values ​​as realistically as possible, we use a benchmark with JMH, the official microbenchmark tool of the OpenJDK community. The aim is to develop a sense of when parallel collectors actually offer added value and under which conditions they may even prove inefficient.

The data model

The benchmark’s data model is based on a practical license management system, such as those typically used in software products, platform services, or license servers. It consists of a compact but expressive Record class called LicenseInfo, which encapsulates all the essential information of a single software license.

At the center is theprecise product allocation across the field productId, which allows conclusions about the licensed application or component to be drawn. Thelicense itself is replaced by a cryptographically unique licenseKey represented, simulated here in the example, by a randomly generated UUID. These two fields are responsible for the formal identification of each license instance.

Two further fields describe temporal aspects: issuedAt indicates the date and time of issuance, and during validUntil marks the expiry time. This not only allows you to check whether a license is currently valid, but also the period for which it was originally created. Based on this period, the average useful life can be calculated, a central metric type for license analyses in companies.

The structure is complemented by the field country, which documents the geographical origin or scope of the license. This is relevant for compliance issues, regional licensing models or sales evaluations. The information is intended as an ISO country abbreviation, but can be flexibly adapted to actual requirements.

Finally, the field isTrial depends on whether it is oneTrial license or a regular license. This boolean field enables differentiated evaluations according to usage type—an important aspect if, for example, a distinction is to be made between active customers and evaluations.

The record definition remains deliberately slim, but provides a solid basis for various analysis methods: from counting and grouping to time series and country statistics. Their immutability and precise semantics make them ideal for use in parallel streams and collecting aggregations. In combination with a user-defined collector, the model allows precise and at the same time high-performance evaluation of extensive license stocks, with full parallelizability.

javaCopy

importjava.time.LocalDateTime;publicrecordLicenseInfo(StringproductId,StringlicenseKey,LocalDateTimeissuedAt,LocalDateTimevalidUntil,Stringcountry,booleanisTrial){}

The test setup

The structure of the benchmark for license analysis follows a systematic pattern, which is typical for microbenchmark-based performance tests with JMH. The aim is to measure under controlled conditions how the running time of a specially developed collector changes as the amount of data increases, both in serial and in parallel execution. The focus is not on comparing different algorithms, but rather on the stream processing strategy’s effect on the aggregation’s overall duration.

The test focuses on the benchmark class LicenseCollectorBenchmark, which is equipped with the annotations of the JMH library. It defines two measuring points: serialCollector() for serial evaluation using stream() and parallelCollector() for parallel evaluation using parallelStream(). Both methods process identical data but use different execution strategies.

Before each benchmark run, the @Setup(Level.Iteration)-annotated method setUp() generates a consistent test data set. This amount of data – 100,000 randomly generated by default LicenseInfo-Instances - is built fresh with each iteration to eliminate possible caching effects or side effects between benchmarks. The values ​​such as product ID, license key, timestamp, country code and test status are generated randomly, creating realistic heterogeneity.

Both benchmark methods then fully aggregate the license data, each using the same custom collector. This collector consolidates the data into one final LicenseStats-Object that summarizes several metrics, including the number of valid licenses, the number of test licenses, the average term, the country-specific distribution and the earliest and latest issuance time. This breadth of aggregation ensures that the collector in the benchmark is sufficiently loaded both computationally and in memory terms.

The actual measurement is carried out via the JMH runtime mechanism with a predefined number of warm-up and measurement cycles. The average execution time per benchmark method is expressed in milliseconds per operation (ms/op), outputting an easily interpretable value representing the total stream processing duration, including accumulation and reduction.

This setup makes it possible to quantify the efficiency advantages – or potential disadvantages – of parallel stream processing under practical conditions. The comparability between serial and parallel is complete because the data is identical, the collector is similar, and only the execution strategy differs.

xmlCopy

import org.openjdk.jmh.annotations.*;import java.time.LocalDateTime;import java.time.Duration;import java.util.*;import java.util.concurrent.*;import java.util.stream.Collector;import java.util.stream.Collectors;import static java.util.concurrent.TimeUnit.MILLISECONDS;@BenchmarkMode(Mode.AverageTime)@OutputTimeUnit(MILLISECONDS)@State(Scope.Thread)public class LicenseCollectorBenchmark { static final int DATA_SIZE = 100_000; List<LicenseInfo> licenseData; @Setup(Level.Iteration) public void setUp() { Random rand = new Random(); licenseData = new ArrayList<>(DATA_SIZE); for (int i = 0; i< DATA_SIZE;i++){licenseData.add(newLicenseInfo("product-"+rand.nextInt(100),UUID.randomUUID().toString(),LocalDateTime.now().minusDays(rand.nextInt(1000)),LocalDateTime.now().plusDays(rand.nextInt(1000)),"country-"+rand.nextInt(20),rand.nextBoolean()));}}@BenchmarkpublicLicenseStatsserialCollector(){returnlicenseData.stream().collect(LicenseCollectors.stats());}@BenchmarkpublicLicenseStatsparallelCollector(){returnlicenseData.parallelStream().collect(LicenseCollectors.stats());}//---datamodel---publicrecordLicenseInfo(StringproductId,StringlicenseKey,LocalDateTimeissuedAt,LocalDateTimevalidUntil,Stringcountry,booleanisTrial){}publicrecordLicenseStats(longtotalValid,longtotalTrial,doubleavgDurationDays,Map<String,Long> byCountry, LocalDateTime oldestIssued, LocalDateTime newestIssued ) {} // --- Collector-Factory --- static class LicenseCollectors { static Collector<LicenseInfo,?,LicenseStats> stats() { class Acc { long valid = 0; long trial = 0; long total = 0; long sumDays = 0; Map<String,Long> byCountry = new ConcurrentHashMap<>(); LocalDateTime oldest = null; LocalDateTime newest = null; void add(LicenseInfo li) { if (li.validUntil().isAfter(LocalDateTime.now())) valid++; if (li.isTrial()) trial++; long days = Duration.between(li.issuedAt(), li.validUntil()).toDays(); sumDays += days; total++; byCountry.merge(li.country(), 1L, Long::sum); if (oldest == null || li.issuedAt().isBefore(oldest)) oldest = li.issuedAt(); if (newest == null || li.issuedAt().isAfter(newest)) newest = li.issuedAt(); } Acc merge(Acc other) { valid += other.valid; trial += other.trial; sumDays += other.sumDays; total += other.total; other.byCountry.forEach((k, v) -> byCountry.merge(k, v, Long::sum)); if (oldest == null || (other.oldest != null&& other.oldest.isBefore(oldest))) oldest = other.oldest; if (newest == null || (other.newest != null&& other.newest.isAfter(newest))) newest = other.newest; return this; } LicenseStats finish() { double avg = total == 0 ? 0 : (double) sumDays / total; return new LicenseStats(valid, trial, avg, byCountry, oldest, newest); } } return Collector.of( Acc::new, Acc::add, Acc::merge, Acc::finish, Collector.Characteristics.UNORDERED, Collector.Characteristics.CONCURRENT ); } }}

Comparison with alternative parallelism concepts

While parallel streams with collectors offer a declarative and comparatively accessible way to use multi-core processors, there are other parallelisation models in the Java platform, each with its own strengths and areas of application.

An obvious comparison is with the ForkJoinPool-API. This allows recursive tasks to be broken down and carried out efficiently in parallel, for example, when searching or sorting large amounts of data. However, unlike streams, fork/join structures are explicit: the developer manually controls the division of tasks and the combination of results. This creates more flexibility, but also more complexity and potential for errors.

Another approach is to use CompletableFuture – especially in combination with methods like supplyAsync() or thenCombineAsync(). The focus here is not on aggregating data but coordinating asynchronous, non-blocking tasks. While streams must have all the available data, they are suitable CompletableFutures for I/O-heavy or delayed processing.

Newer Java versions also provide structures such as StructuredTaskScope, which comes into play - a still experimental but promising approach to organise parallel tasks better and explicitly control their lifespan. Unlike streams, this API allows structured concurrency with clearly defined error handling and cancellation semantics.

Parallel collectors position themselves within this spectrum as a compact, declarative mechanism focusing on data-centric, memory-bound processing. They offer low barriers to entry, but are limited in terms of expressiveness and controllability. They reach their natural limits in more complex scenarios - with dependencies between subtasks, adaptive scheduling or concurrent I/O operations.

Therefore, the choice of the appropriate parallelism model should always be based on the task’s requirements: Where pure data aggregation is involved, streams and parallel collectors are ideal. However, where control flow, resilience, or the ability to integrate into distributed systems is required, other APIs are more effective.

Conclusion and outlook

Parallel collectors are not a sideshow to the Java Streams API - they are at the heart of whether modern computing can truly scale. Anyone who understands their structure quickly realizes that the correct aggregation in a parallel context is not a coincidence, but the result of precise planning. Associativity, deterministic processing and thread-safe data structures are not academic footnotes, but central pillars of this architecture.

The concepts shown here—from the structural basis to best practices to your own implementations—are intended not only to convey the functionality but also to create awareness of the responsibility that comes with parallelism. What is harmless in a single-threaded scenario can lead to serious errors in a parallel environment.

The question for the future is how the Streams API will continue to develop. More structured parallelism models, which are based on concepts such as structured concurrency or data flow graphs, are conceivable. Integration with reactive streams or declarative parallelism à la Fork/Join-DSL would also be a logical next step.

One thing is certain: Anyone who learns to use parallel collectors correctly and efficiently today will lay the foundation for high-performance and scalable data processing in the Java platform of tomorrow.

Happy Coding

Sven

]]>JavaStreamshttps://svenruppert.com/posts/dns-attacks-explained/Mon, 07 Apr 2025 08:48:06 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/dns-attacks-explained/1. Getting started – trust in everyday internet life Anyone who enters a web address like “www.example.de” into the browser expects a familiar website to appear within seconds. Whether in the home office, at the university, or in the data center, access to online services is now a given. The underlying technical processes are invisible to most users; even in IT practice, they are often taken for granted. One of these invisible processes is name resolution by the Domain Name System (DNS).<![CDATA[

1. Getting started – trust in everyday internet life

Anyone who enters a web address like “www.example.de” into the browser expects a familiar website to appear within seconds. Whether in the home office, at the university, or in the data center, access to online services is now a given. The underlying technical processes are invisible to most users; even in IT practice, they are often taken for granted. One of these invisible processes is name resolution by the Domain Name System (DNS).

1. 1. Getting started – trust in everyday internet life
2. 2. The Domain Name System – backbone of the digital world
3. 3. Invisible Attack Surface – Why DNS is vulnerable
4. 4. DNS cache poisoning – The classic attack
5. 5. Amplification and Deception – DNS as a vehicle for DDoS and tunneling
6. 6. DNS Hijacking – When attackers dictate the path
7. 7. Attack detected – but how?
8. 8. Protective measures – think and implement DNS securely
   1. 8.1 General Network Protection Measures
   2. 8.3 Advanced protection measures at protocol and application level
9. 9. Conclusion – trust needs protection

DNS is the phone book of the Internet. It ensures that human-readable domain names are translated into numeric IP addresses so that computers worldwide can communicate with each other. Without DNS, there would be no URLs, emails, or APIs. Nevertheless, DNS has had a shadowy existence in the security discourse for a long time. It is not a purely technical detail but rather a central link in the trust relationship between the user, application, and infrastructure.

Because DNS works in the background, it is a popular target for attackers. DNS attacks aim to disrupt, manipulate, or redirect this process for your own purposes, with potentially serious consequences. Anyone who relies on a domain name always leading to the correct IP address risks falling into a well-disguised trap. The affected applications appear to continue to work perfectly—they just communicate with the wrong counterpart.

This text highlights the diverse forms of attacks on DNS, how they work, and their implications for the operation and development of secure systems. Particular attention is paid to the question of how DNS must be considered from the perspective of modern applications, especially in Java, so as not to become the Achilles heel of an otherwise solid security architecture.

2. The Domain Name System – backbone of the digital world

The Domain Name System has become integral to today’s Internet communication. It was developed in the early 1980s to replace the increasingly confusing web of growing hostnames with a scalable, hierarchical structure. Instead of remembering IP addresses, people can now use spoken domain names, and rely on them reliably pointing to the correct servers.

The basic job of DNS is to translate a string like “www.uni-heidelberg.de“ to an IP address like “129.206.100.168”. This process occurs in several steps and is handled by different types of servers. The process usually begins with the so-called resolver – a system service or external service provider that accepts requests from end devices. This resolver in turn queries root name servers, top-level domain servers (e.g. for .de, .com, .org) and finally authoritative name servers until it finds the IP address it is looking for.

The concept of caching is essential to DNS’s efficiency. Each resolver stores already resolved queries for a defined period of time to speed up re-queries. Operating systems and applications can also maintain their own DNS caches. This behavior contributes significantly to reducing global DNS traffic, but it is also one of the biggest weak points, as later chapters will show.

DNS is based on the User Datagram Protocol (UDP) and is stateless and unencrypted by design. Although extensions such as DNSSEC or DNS-over-HTTPS exist, they are far from being used widely in everyday life. This means that DNS remains an open system, efficient but vulnerable. The assumption that DNS “just works” quickly becomes a dangerous illusion from a security perspective.

3. Invisible Attack Surface – Why DNS is vulnerable

The architecture of the Domain Name System follows the paradigm of openness. This means that requests are standardised, unencrypted, and mostly stateless, which is precisely why they are particularly high-performance. But the very properties that make DNS a robust part of the Internet infrastructure also pose serious risks. The vulnerability of the DNS is not an accidental by-product of modern forms of attack but is rooted in the original system design.

DNS was designed when the Internet was a trusted space characterised by academic collaboration and open standards. Security mechanisms were not part of the design goal at the time. Accordingly, DNS still lacks built-in integrity, authenticity, and confidentiality. Queries are not verified, answers are not signed, and the source is not checked. This creates attack vectors that are still being exploited today by actors of all stripes.

Additionally, DNS requests are typically transmitted over UDP, which makes them easy to forge or intercept. Transaction IDs and ports offer minimal protection against spoofing, but these hurdles are technically easy to overcome with today’s means. Even with TCP-based fallbacks, absolute protection is lacking without additional measures such as DNSSEC.

A particular vulnerability lies in the caching behavior of resolvers and applications. Once saved, answers are considered valid for a certain period, regardless of whether they were correct or manipulated. Attackers can specifically exploit this behavior to inject false answers into the cache and thus redirect many subsequent requests. This so-called cache poisoning is one of the oldest but still most effective attacks on the DNS.

DNS is therefore not a secure anchor of trust, at least not without additional measures. Anyone using DNS in a security-critical architecture, such as modern Java applications, cloud APIs or container infrastructures, should know the inherent weaknesses. DNS is robust, but it can also be manipulated. This makes it an attractive target – and an often underestimated attack surface.

4. DNS cache poisoning – The classic attack

Among the numerous attack methods on the domain name system, DNS cache poisoning—also known as DNS spoofing—is considered particularly perfidious. The aim of this attack is to manipulate the caching of DNS responses. The attacker injects a fake DNS response into a resolver’s cache, with the effect that all subsequent queries to a specific domain name point to a false IP address. The affected users usually do not notice the attack: the website looks as usual, except that the attacker controls it.

The process of such an attack is as technically interesting as it is security-relevant. The attacker does not wait for a resolver to resolve a specific domain but proactively tries to outsmart the resolver with fake answers. To do this, it must guess the transaction ID of the DNS request - a short, 16-bit random number - and respond faster than the legitimate DNS server. If this succeeds, the resolver accepts the incorrect answer as valid and stores it in the cache.

What is particularly critical is that this manipulation persists: As long as the cache entry remains valid (TTL), every new request will return the same fake answer. This way, individual users and entire organisations can be systematically misled - for example, on phishing sites, fake authentication services or malware sources. This can result in a REST API being accessed supposedly successfully in Java applications, even though a completely different infrastructure is responding in the background.

Modern DNS implementations now rely on additional protection mechanisms such as random source ports, rate limiting, or DNSSEC. But many resolvers—especially on local networks or outdated systems—remain vulnerable. Even browsers or libraries with built-in DNS caching can become targets.

For developers, this means that the trust assumption that InetAddress.getByName() always returns the “correct” address is dangerous. Explicit protective measures are required when DNS is used in security-critical contexts, for example, through the use of cryptographically signed DNS responses (DNSSEC), deliberate cache invalidation, or explicit validation of the remote site, for example, via TLS certificates.

DNS cache poisoning is a lesson in how a helpful performance optimization—caching—can turn into the exact opposite. Anyone who wants to operate or use DNS securely must understand how the system works and how easily it can be manipulated at its core.

5. Amplification and Deception – DNS as a vehicle for DDoS and tunneling

In addition to the targeted manipulation of DNS caches, there are other attack scenarios in which DNS serves as a technical vehicle for higher-level goals, such as denial-of-service attacks or secret data transmission. Two particularly relevant variants in this context are DNS amplification and DNS tunneling.

A DNS amplification attack is a form of reflected denial of service (DoS) that exploits the asymmetric structure of the DNS protocol. The attacker sends a small DNS query - often of type ANY - to an open resolver, but spoofs the source address to point to the victim. The resolver responds with a significantly larger amount of data than the request, which is sent directly to the victim. Multiplied across many resolvers, this creates a massive data stream that is explicitly used to overload servers. DNS amplification is so effective because the attacker can cause disproportionate damage with minimal use of resources.

Another gateway is DNS tunneling. This technique uses DNS packets to pass any other data through firewalls or security measures. Since DNS traffic is hardly restricted in many networks, the protocol is ideal for bypassing control authorities. The payload—such as commands to malware or stolen data—is hidden in seemingly legitimate DNS queries or responses. On the receiving end, this information is extracted and evaluated again. DNS tunneling is considered particularly insidious because it is difficult to detect using traditional security tools.

A simple Java example demonstrating how a DNS tunnel could be hidden in a seemingly legitimate HTTP communication can be implemented using Java SE’s on-board tools. In the following scenario, we implement a minimalist REST server with com.sun.net.httpserver.HttpServer, which receives DNS-like encoded data, representative of a possible C2 channel (Command & Control).

HttpServer server = HttpServer.create(new InetSocketAddress(8080), 0);

server.createContext("/api/lookup", exchange - > {

** if (“GET”.equals(exchange.getRequestMethod())) {**

** URI requestURI = exchange.getRequestURI();**

** String query = requestURI.getQuery();**

** if (query != null && query.contains(“q=”)) {**

** String domainPayload = query.split(“q=”)[1];**

** String decoded = domainPayload.replace(".", " “); // simplified payload decoding**

** System.out.println("[DEBUG] Empfangenes DNS-Tunnel-Payload: " + decoded);**

** }**

** String response = “OK”;**

** exchange.sendResponseHeaders(200, response.length());**

** exchange.getResponseBody().write(response.getBytes());**

** exchange.close();**

** }**

});

server.setExecutor(null);

server.start();

In this simplified simulation, a DNS-like query parameter is received via a REST interface - e.g. b. http://localhost:8080/api/lookup?q=cmd.run.download. The server “interprets” the dot as a separator of an encoded DNS structure, as is common in DNS tunneling. In a real attack variant, this mechanism would be used by malware or scripts for bidirectional data exchange. What is critical here is that the transport takes place over seemingly unsuspicious channels - HTTP, DNS, ICMP - while the actual payload is hidden deep in protocol structures.

Both types of attacks have in common that DNS is not a direct target, but rather a tool - a transport system for superimposed attack targets. This results in the need for administrators and developers to monitor DNS traffic functionally and from a security perspective. In Java-based architectures, DNS dependencies should be consciously modeled, logged and – where possible – checked for validation and integrity. A simple forwarding of data via DNS may seem harmless, but it may be the start of a widespread attack.

6. DNS Hijacking – When attackers dictate the path

While DNS cache poisoning focuses on manipulating cached responses, DNS hijacking targets the source itself: the DNS resolver or the infrastructure that controls it. The attack begins when users resolve their domain names - either on the local device, on the home router, on corporate networks, or via an ISP’s resolver.

With DNS hijacking, DNS requests are systematically redirected - for example, through manipulation of network settings, compromised firmware, malicious DHCP servers or malware that specifically changes the configuration of a system. The resolver then no longer returns the IP addresses expected by the user, but instead forwards requests to servers controlled by the attacker. In practice, this means: Even if a user “www.bank.de“ enters something in your browser, you don’t end up on the real online banking portal, but on a phishing page that looks real.

This attack is hazardous because it is often difficult to detect. Unlike classic phishing, which relies on fake links or typos, DNS hijacking makes the actual URL appear completely legitimate. The certificate can also appear valid under certain circumstances—for example, with local man-in-the-middle proxies or if the attacker manages to obtain a certificate via compromised certificate authorities.

Java applications are also potentially affected, primarily if they communicate dynamically with external services. If the host machine’s DNS configuration is compromised, all DNS lookups within the JVM are affected, from REST clients to LDAP connections to email components. The application often does not recognize the problem because the connection is technically established correctly, just to the wrong destination.

The only solution is to take measures that secure DNS on several levels: through fixed resolver configurations, validation of the other side via TLS, use of secure DNS protocols such as DNS-over-HTTPS (DoH) or DNS-over-TLS (DoT), and by monitoring for suspicious name resolutions. In security-critical Java applications, it is recommended to compare the resolved IP addresses with allowlists or even to forego DNS queries if the target systems are known.

DNS hijacking is a remarkably sophisticated form of deception because it attacks trust where it seems most evident: in the integrity of the technical infrastructure itself.

7. Attack detected – but how?

Detecting DNS attacks is one of the most challenging tasks in network security. This is primarily because DNS is a widespread, dynamic and often invisible protocol. Name resolution occurs in the background, usually without direct user contact, and is often only incompletely recorded in logs and monitoring systems. Accordingly, many DNS-based attacks remain undetected for a long time, mainly when they are carried out precisely and subtly.

A central problem: Neither the browser nor the application registers whether a DNS response was legitimate, as long as the response is formally correct. A manipulated result usually does not differ from a real one, unless it is actively checked. This is where anomaly detection methods come into play. They try to identify unusual behavior, such as sudden changes in the IP assignment of known domains, an accumulation of DNS responses with short TTL or unusual destination addresses outside of usual networks.

Intrusion detection systems (IDS) and DNS-specific monitoring solutions such as Zeek, OpenDNS or SecurityOnion can help detect such anomalies. However, the prerequisite is that DNS traffic is completely recorded and evaluated, which brings additional complexity with encrypted DNS (DoH/DoT). In controlled networks, it is therefore recommended to route DNS via dedicated resolvers that can be logged and monitored.

In Java applications, suspicious name resolutions can be detected by logging IP mappings, comparing against expected hostnames, or active whitelisting. Metrics such as the number of different IP addresses per domain over time or the reuse of certain, unexpected IP ranges are also valuable information.

In the long term, comprehensive validation of DNS responses via DNSSEC is essential in ensuring integrity. However, DNSSEC is not retroactive - it only protects when domain owners and resolvers actively work together. Accordingly, awareness of DNS attacks and their detection is a crucial component of any security strategy: Anyone who only sees DNS as a technical infrastructure risks being compromised exactly where trust begins - with name resolution.

8. Protective measures – think and implement DNS securely

8.1 General Network Protection Measures

Detecting DNS attacks is one of the most challenging tasks in network security. This is primarily because DNS is a widespread, dynamic and often invisible protocol. Name resolution occurs in the background, usually without direct user contact, and is often only incompletely recorded in logs and monitoring systems. Accordingly, many DNS-based attacks remain undetected for a long time, especially when they are carried out specifically and subtly.

A central problem: Neither the browser nor the application registers whether a DNS response was legitimate, as long as the reaction is formally correct. A manipulated result usually does not differ from a real one, unless it is actively checked. This is where anomaly detection methods come into play. They try to identify unusual behavior, such as sudden changes in the IP assignment of known domains, an accumulation of DNS responses with short TTL or unusual destination addresses outside of usual networks.

Intrusion detection systems (IDS) and DNS-specific monitoring solutions such as Zeek, OpenDNS or SecurityOnion can help detect such anomalies. However, the prerequisite is that DNS traffic is completely recorded and evaluated, which brings additional complexity with encrypted DNS (DoH/DoT). In controlled networks, it is therefore recommended to route DNS via dedicated resolvers that can be logged and monitored.#### 8.2 Security within Java applications

In Java applications, suspicious name resolutions can be detected by logging IP mappings, comparing against expected hostnames, or active whitelisting. Metrics such as the number of different IP addresses per domain over time or the reuse of certain, unexpected IP ranges are also valuable information.

In the long term, comprehensive validation of DNS responses via DNSSEC is essential in ensuring integrity. However, DNSSEC is not retroactive - it only protects when domain owners and resolvers actively work together. Accordingly, awareness of DNS attacks and their detection is a crucial component of any security strategy: Anyone who only sees DNS as a technical infrastructure risks being compromised exactly where trust begins - with name resolution.

8.3 Advanced protection measures at protocol and application level

Various protection measures can be implemented for Java applications to minimize the risks of DNS-based attacks. A key approach is securing the name resolution process within the JVM. The class InetAddress, for example, uses resolvers close to the operating system, which makes the JVM fundamentally vulnerable to manipulation at the system level. In security-critical applications, it can, therefore, make sense to implement your own resolvers with DNSSEC validation or to use external, verified services.

Additionally, it is recommended to be combined with transport encryption and hostname validation at the application level. The HttpClient from the java.net.http module in Java 11+ enables e.g. B. direct control over TLS certificate checking. By securing via public key pinning, certificate transparency logs or targeted whitelists, manipulated DNS responses can be detected and blocked.

Another security option is to use alternative DNS protocols such as DoH (DNS over HTTPS) or DoT (DNS over TLS) via upstream proxies. These can also be integrated into containerized environments such as Kubernetes as sidecar resolvers. A locally running DNS proxy (e.g., CoreDNS with DNSSEC and DoH support) can be used as a trustworthy source within the application.

A practical example of a hedging measure in Java is using your own X509TrustManager to not rely solely on the system-provided chain of trust for the TLS connection. This is particularly useful if you want to counter DNS manipulations that do not break the TLS channel but subvert a false certificate chain - for example, through a compromised CA certificate or a manipulated trust store.

TrustManager[] trustManagers = new TrustManager[] {

** new X509TrustManager() {**

** public X509Certificate[] getAcceptedIssuers() {**

** return new X509Certificate[0];**

** }**

** public void checkClientTrusted(X509Certificate[] certs, String authType) {**

** // Application-specific testing – e.g. E.g. fingerprinting or pinning**

** }**

** public void checkServerTrusted(X509Certificate[] certs, String authType) throws CertificateException {**

** for (X509Certificate cert : certs) {**

** cert.checkValidity();**

** if (!“CN=trusted.example.com”.equals(cert.getSubjectDN().getName())) {**

** throw new CertificateException(“Untrusted CN: " + cert.getSubjectDN());**

** }**

** }**

** }**

** }**

};

SSLContext sslContext = SSLContext.getInstance(“TLS”);

sslContext.init(null, trustManagers, new SecureRandom());

HttpClient client = HttpClient.newBuilder()

** .sslContext(sslContext)**

** .build();**

This example deliberately checks the subject of the server certificate passed. This makes it possible to ensure that the remote site actually has the expected identity, even if a DNS hijacker manipulates the target system. A more detailed check would be necessary in a productive system, for example, via public key pinning, dedicated trust stores or certificate chain validation.

Last but not least: DNS requests should be logged, checked for plausibility and ideally compared with an allowlist. This allows a simple but effective protective layer to be implemented, especially in applications where external communication is security-critical.

9. Conclusion – trust needs protection

Given the central role of the domain name system in virtually every form of modern network communication, it is remarkable how long DNS security played a secondary role in the architecture of many applications. The forms of attack highlighted here - from cache poisoning to DNS hijacking to tunneling and DDoS - clearly show that DNS is much more than a harmless auxiliary component. It is one of the most critical infrastructures of all, and simultaneously one of the most vulnerable.

Anyone who develops applications today that rely on Internet communication must think about DNS functionally and in terms of security. This applies to web portals, microservices, container environments, and IoT devices. In the Java ecosystem, in particular, the platform from version 11 provides numerous APIs and mechanisms to specifically secure DNS dependencies, be it through explicit TLS configurations, validation of target addresses, logging of suspicious resolutions, or the use of alternative resolvers.

DNS is not a static problem. The attack vectors continue to develop, and with them, the requirements for protective mechanisms. At a time when trust is becoming a currency of digital interactions, DNS can no longer remain a blind spot. It is time for DNS security to be established as an integral part of every security-conscious software and infrastructure architecture - in the code, operations and all stakeholders’ minds. logged, checked for plausibility and ideally compared with an allowlist. This allows a simple but effective protective layer to be implemented, especially in applications where external communication is security-critical.

Happy Coding

Sven

]]>Securityhttps://svenruppert.com/posts/java-cryptography-architecture-jca-an-overview/Thu, 03 Apr 2025 12:22:30 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/java-cryptography-architecture-jca-an-overview/The Java Cryptography Architecture (JCA) is an essential framework within the Java platform that provides developers with a flexible and extensible interface for cryptographic operations. It is a central component of the Java Security API and enables platform-independent implementation of security-critical functions.<![CDATA[

TheJava Cryptography Architecture (JCA) is an essential framework within the Java platform that provides developers with a flexible and extensible interface for cryptographic operations. It is a central component of the Java Security API and enables platform-independent implementation of security-critical functions.

At its core, the JCA provides mechanisms for various cryptographic applications, including the calculation of hashes to ensure data integrity, the generation and verification of digital signatures, and methods for encrypting and decrypting sensitive information. Supporting both symmetrical and asymmetrical encryption methods, it ensures a high level of security when processing data. Cryptographic key management is another key aspect that includes secure storage and exchange of keys.

1. How is the JCA integrated into the Security API?
2. Symmetric encryption
3. The first practical steps
4. Asymmetric encryption
5. A practical example
6. Digital signatures
7. A practical example
8. Now let’s combine symmetric and asymmetric encryption

A significant feature of the JCA is the ability to integrate different security providers (so-called providers) that implement different cryptographic algorithms. In addition to the providers that come standard with Java, such as SunJCE, there are alternative implementations, including open source libraries such as Bouncy Castle and hardware-based solutions, that can be integrated via PKCS#11 interfaces.

The JCA’s architecture is characterised by a modular design that allows a separation between the API and the concrete implementations. New cryptographic algorithms can be added via the so-called Service Provider Interface (SPI) without having to adapt existing application code, contributing to the long-term maintainability and security of Java applications.

In addition to classic cryptography, the JCA also supports mechanisms for the secure generation of random numbers, the calculation of Message Authentication Codes (MACs) to ensure data integrity, and the management of certificates and trust stores. In particular, integration with the Java KeyStore API offers a comprehensive basis for the secure handling of X.509 certificates and cryptographic keys.

How is the JCA integrated into the Security API?

TheJava Security API is a comprehensive framework for implementing security-related mechanisms in Java applications. It provides various functions that cover various aspects of IT security, including cryptography, authentication, authorization, network security, and cryptographic key management. A central component of this framework is theJava Cryptography Architecture (JCA).

While JCA is primarily responsible for the cryptographic functions, it extends theJava Cryptography Extension (JCE). This concept includes algorithms for symmetric encryption, key agreement and block cipher modes. In addition to cryptography, the Java Security API includes other security-relevant modules. TheJava Secure Socket Extension (JSSE) API enables secure communication via network protocols such as TLS and SSL, enabling encrypted connections for web applications and other network services to be realised. This provides authentication and authorisationJava Authentication and Authorization Service (JAAS) API mechanisms available to verify user identities and control access rights. This architecture allows integration with various authentication sources such as user-password combinations, Kerberos or biometric methods.

Another essential component of the framework is theJava Public Key Infrastructure (PKI) API , which deals with the management of digital certificates and public keys. This API plays a central role in the realization of TLS encryption and the implementation of digital signatures. Certificate management is carried out via so-called KeyStores, which enable secure storage of cryptographic keys. In addition, the framework supports digital signatures for XML documents through theJava XML Digital Signature API (XMLSig) , which is particularly used in web services and SAML authentication scenarios.

The distinction between the Java Security API and the JCA lies in their respective focus. While the Java Security API provides a comprehensive collection of security-related mechanisms and covers both cryptographic and non-cryptographic security aspects such as authentication, permission management and network security, the JCA focusesexclusively on providing cryptographic operations. In this context, JCA offers basic mechanisms for encryption and digital signatures, but JCE is required for modern symmetric encryption methods. Likewise, although JCA includes basic functions for managing cryptographic keys, more comprehensive management of certificates and public key infrastructure requires the use of the PKI API.

In practice, this means that developers access the JCA and JCE directly to implement cryptographic security mechanisms, while specific modules such as JAAS or JSSE are used for higher security-related functions such as authentication, authorisation or the management of secure network connections. The Java Security API thus forms a coherent but modular security architecture that can be used specifically depending on the application.

In this article we will only deal with the JCA. The other functional units will be gradually added in later articles.

Symmetric encryption

The symmetric encryption within theJava Cryptography Architecture (JCA) offers a powerful and flexible interface for the secure processing of confidential data. It is based on the use of a single secret key for encryption and decryption, which makes it more efficient and faster than asymmetric methods. The JCA provides the Cipher-Class that acts as a central mechanism for symmetric cryptography and enables the implementation of various block and stream ciphers.

A key feature of symmetric encryption in the JCA is its support for modern block ciphers such asAES (Advanced Encryption Standard) , which is considered the current industry standard, as well as older methods such asDES (Data Encryption Standard) and its improved version,Triple DES (3DES). The API allows the use of various operating modes, includingECB (Electronic Codebook) , which is considered unsafe, as well as safer modes such asCBC (Cipher Block Chaining) ,CFB (Cipher Feedback Mode) andGCM (Galois/Counter Mode). GCM in particular is preferred because, in addition to confidentiality, it also offers integrity protection through Authenticated Encryption (AEAD).

A central concept of the JCA is the separation between the abstract API and the concrete implementations provided by various cryptography providers. This keeps the actual implementation interchangeable without tying the application to a specific algorithm or library. The initialization of a Cipher-Object is done by specifying the algorithm as well as the desired operating mode and an optional padding method that ensures block-by-block processing of the data. Common padding schemes arePKCS5Padding andNoPadding , the latter requiring manual adjustment of the input data.

The JCA also provides mechanisms for the secure generation and management of keys. About the KeyGenerator-Class, symmetrical keys can be generated based on a defined key length, with key lengths of 128, 192 and 256 bits being particularly common for AES. To ensure the security of the keys, they can be passed through theJava KeyStore (JKS) or external hardware security modules (HSMs). Additionally, the API allows keys to be imported and exported in standardised formats to ensure interoperability with other cryptographic systems.

Another central element of symmetric encryption in the JCA is the treatment of initialisation vectors (IVs), which are crucial for the security of the encryption, especially in modes such as CBC or GCM. IVs must be unique for each encryption operation to prevent attacks such as replay or chosen plaintext attacks. In practice, IVs are often over SecureRandom generated to ensure high entropy and stored or transmitted along with the encrypted data.

The architecture of the JCA allows developers to use symmetric encryption not only for straightforward file and message encryption, but also in more complex scenarios such asTransport Layer Security (TLS) ,Database encryption andHardware-assisted encryption. By combining it with other JCA components, such as theMac class for Message Authentication Codes (e.g. HMAC-SHA256) orKey Derivation Functions (e.g. PBKDF2) , additional security features such as authentication and key derivation can be implemented.

The first practical steps

Let’s now move on to a simple example of using symmetric encryption within the Java Cryptography Architecture (JCA) and use the AES algorithm family in combination with a secure operating mode and padding. The implementation relies on the Cipher-Class, a central interface in Java for processing cryptographic operations. Choosing secure key management is crucial, which is why the key is stored using the KeyGenerator class, which is created. SecureRandom is used to ensure high entropy and encryption security for the initialisation data.

Within the method, a character array (char[]) is used instead of a string to ensure that sensitive data does not remain in the string pool and can be specifically overwritten after processing.(Here, I refer to the previous article, in which I described this in more detail.) After the key and IV initialisation, encryption is carried out using a Cypher object in AES-CBC mode with PKCS5Padding. The encrypted string is then stored in a byte array. Decryption is done in reverse order by reinitialising the Cipherobject with the same key and IV, which allows the original characters to be restored from the encrypted data stream.

The following source code demonstrates the implementation of this concept:

javaCopy

importjavax.crypto.Cipher;importjavax.crypto.KeyGenerator;importjavax.crypto.SecretKey;importjavax.crypto.spec.IvParameterSpec;importjava.security.SecureRandom;importjava.util.Arrays;publicclassSymmetricEncryptionExample{publicstaticvoidmain(String[]args)throwsException{KeyGeneratorkeyGenerator=KeyGenerator.getInstance("AES");keyGenerator.init(256);SecretKeysecretKey=keyGenerator.generateKey();SecureRandomsecureRandom=newSecureRandom();byte[]iv=newbyte[16];secureRandom.nextBytes(iv);IvParameterSpecivSpec=newIvParameterSpec(iv);char[]originalText={'H','e',​​'l','l','o',' ','J','C','A'};byte[]encryptedText=encrypt(originalText,secretKey,ivSpec);char[]decryptedText=decrypt(encryptedText,secretKey,ivSpec);System.out.println(decryptedText);Arrays.fill(decryptedText,'\0');}privatestaticbyte[]encrypt(char[]input,SecretKeykey,IvParameterSpecivSpec)throwsException{Ciphercipher=Cipher.getInstance("AES/CBC/PKCS5Padding");cipher.init(Cipher.ENCRYPT_MODE,key,ivSpec);returncipher.doFinal(newString(input).getBytes());}privatestaticchar[]decrypt(byte[]input,SecretKeykey,IvParameterSpecivSpec)throwsException{Ciphercipher=Cipher.getInstance("AES/CBC/PKCS5Padding");cipher.init(Cipher.DECRYPT_MODE,key,ivSpec);byte[]decryptedBytes=cipher.doFinal(input);char[]decryptedChars=newchar[decryptedBytes.length];for(inti=0;i<decryptedBytes.length;i++){decryptedChars[i]=(char)decryptedBytes[i];}returndecryptedChars;}}

Asymmetric encryption

Theasymmetric encryption within the Java Cryptography Architecture (JCA) provides a flexible and secure way to encrypt and decrypt data using two different keys: a public key for encryption and a private key for decryption. This method is based on one-way mathematical functions, which make it practically impossible to calculate the private key from the public key. This makes asymmetric cryptography particularly suitable for applications that require secure key distribution and digital signatures, including Transport Layer Security (TLS), Public Key Infrastructure (PKI), and various authentication mechanisms.

The JCA provides with the Cipherclass provides a central API that enables the implementation of asymmetric encryption methods. The algorithms are particularly widespreadRSA (Rivest-Shamir-Adleman) ,Elliptic Curve Cryptography (ECC) as well asDiffie-Hellman for key exchange. RSA is based on the factorization of large prime numbers and is widely used as a classic public key method. In contrast, ECC uses elliptic curves over finite fields and offers comparable security with a smaller key length, making it particularly preferred for resource-constrained environments such as mobile devices or smart cards.

The key is generated via the KeyPairGenerator-Class that allows to generate asymmetric key pairs with different key lengths. The choice of key length affects security and computing power, with RSA typically operating at 2048 or 4096 bits, while elliptic curves use more efficient 256 or 384 bit keys. The public key generated can be freely distributed, while the private key must be kept secure, for example by storing it in aJava KeyStore (JKS) or a hardware security solution such as a Hardware Security Module (HSM).

A Cipher object is initialised with the desired algorithm and mode for the encryption. The JCA supports various padding mechanisms necessary to adapt input data to the block size of the encryption function used. At RSA,PKCS1Padding andOAEP (Optimal Asymmetric Encryption Padding) are standard methods, with OAEP being preferred due to its increased security, as it protects against adaptive chosen ciphertext attacks. Due to its high computational complexity, asymmetric encryption is primarily suitable for encrypting small amounts of data, such as key material or hash values, rather than large files or streaming data.

Another central application area of ​​asymmetric cryptography in the JCA is the digital signature, which is carried out via the Signature class. The private key creates a signature for a message or file, which can later be verified using the public key. Digital signatures ensure both the authenticity and integrity of the transmitted data by ensuring that the message comes from the specified sender and has not been subsequently altered. In particular, algorithms likeRSA with SHA-256 ,ECDSA (Elliptic Curve Digital Signature Algorithm) orEdDSA (Edwards-Curve Digital Signature Algorithm) are used in modern applications to ensure authenticity and data integrity.

In addition to classic asymmetric encryption, the JCA also supports hybrid methods in which asymmetric cryptography is used to securely transmit a symmetric key, which is then used for data encryption. This procedure is used, for example, TLS (Transport Layer Security) or PGP (Pretty Good Privacy), to combine the advantages of both types of cryptography: the security of asymmetric key distribution and the efficiency of symmetric encryption.

The JCA’s architecture allows developers to flexibly use asymmetric encryption schemes by supporting a variety of security vendors that provide different implementations. By default, the Java platform offers support for RSA, DSA and elliptic curves, while alternative providers such asBouncy Castle enable additional algorithms and optimisations.

A practical example

This example uses the RSA algorithm to implement asymmetric encryption, encrypting data with a public key and decrypting it with the corresponding private key. The implementation is done using the Cipher class. The KeyPairGenerator class is used to generate the key pair, which makes it possible to generate an RSA key with a length of 2048 bits. The resulting key pair consists of a public key used for encryption and a private key required for decryption.

The encryption method takes char[] as input to ensure that no sensitive data is included as String remains in memory. The characters are explicitly converted to a byte array before encryption to enable direct storage operations. Afterwards, Cipherobject is put into encryption mode with the public key, and the conversion is performed. Once encryption is complete, the original byte array is securely overwritten to prevent potential reconstruction in memory.

Decryption is carried out using an analogous procedure. The Cipher object is initialised with the private key to return the previously encrypted byte array to its original state. After decryption, the byte array turns back into a char[] converted without doing a String conversion. To protect the decrypted data, the original and decrypted byte arrays are overwritten with null values ​​after processing.

javaCopy

importjavax.crypto.Cipher;importjava.security.KeyPair;importjava.security.KeyPairGenerator;importjava.security.PrivateKey;importjava.security.PublicKey;importjava.util.Arrays;publicclassAsymmetricEncryptionExample{publicstaticvoidmain(String[]args)throwsException{KeyPairGeneratorkeyPairGenerator=KeyPairGenerator.getInstance("RSA");keyPairGenerator.initialize(2048);KeyPairkeyPair=keyPairGenerator.generateKeyPair();PublicKeypublicKey=keyPair.getPublic();PrivateKeyprivateKey=keyPair.getPrivate();char[]originalText={'H','e',​​'l','l','o',' ','J','C','A'};byte[]encryptedText=encrypt(originalText,publicKey);char[]decryptedText=decrypt(encryptedText,privateKey);System.out.println(decryptedText);Arrays.fill(decryptedText,'\0');Arrays.fill(originalText,'\0');}privatestaticbyte[]encrypt(char[]input,PublicKeykey)throwsException{Ciphercipher=Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");cipher.init(Cipher.ENCRYPT_MODE,key);byte[]inputBytes=newbyte[input.length];for(inti=0;i<input.length;i++){inputBytes[i]=(byte)input[i];}byte[]encryptedBytes=cipher.doFinal(inputBytes);Arrays.fill(inputBytes,(byte)0);returnencryptedBytes;}privatestaticchar[]decrypt(byte[]input,PrivateKeykey)throwsException{Ciphercipher=Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");cipher.init(Cipher.DECRYPT_MODE,key);byte[]decryptedBytes=cipher.doFinal(input);char[]decryptedChars=newchar[decryptedBytes.length];for(inti=0;i<decryptedBytes.length;i++){decryptedChars[i]=(char)decryptedBytes[i];}Arrays.fill(decryptedBytes,(byte)0);returndecryptedChars;}}

Digital signatures

Digital signatures are cryptographic mechanisms that ensure digital messages or documents’ authenticity, integrity and non-repudiation. They are based on asymmetric cryptography and use a key pair consisting of a private key used to generate the signature and a public key used for verification. The central idea behind digital signatures is to confirm the sender’s identity and ensure that the transmitted data has not been tampered with during transmission or storage.

Creating a digital signature begins with calculating a hash value of the message to be signed using a cryptographic hash function. This hash value represents a unique, compact representation of the original message, with even a minimal change to the message resulting in a completely different hash. This hash value is then encrypted with the sender’s private key, creating the digital signature. This signature is transmitted or stored along with the original message to ensure its authenticity later.

The recipient verifies the digital signature using the sender’s public key. To do this, the received message is first hashed again to calculate a new hash value. In parallel, the digital signature is decrypted using the public key, restoring the original hash value generated by the sender. If both hash values ​​match, the recipient can assume with a high degree of certainty that the message is authentic and has not been changed. If the hash values ​​are different, it means either that the message has been tampered with or that the signature was created with a different key, indicating an unauthorised modification or a forged identity.

The security of digital signatures depends largely on the strength of the underlying mathematical procedures and the secure handling of the private keys. Classic signature algorithms likeRSA orDSA (Digital Signature Algorithm) are increasingly being used by more modern processes such asECDSA (Elliptic Curve Digital Signature Algorithm) orEdDSA (Edwards-Curve Digital Signature Algorithm) , as they offer comparable security with shorter keys and higher efficiency. The continuous development of cryptographic standards is essential to keep digital signatures secure in the long term against future threats, such as those from quantum computers.

A practical example

This digital signature implementation example demonstrates the creation and verification of a signature using theRSA signature algorithm with SHA-256. The Signature-Class carries out the implementation. To generate a valid key pair, the KeyPairGenerator class makes it possible to generate an RSA key pair with a length of 2048 bits. The resulting key pair consists of a private key, which is used to create the signature, and a public key, which is used for later verification.

Signature generation begins by processing an input known as char[] that is present. Before signing, the characters are converted directly into a byte array without creating strings. The Signature instance is initialised with the private key, after which the data is processed and the signature is generated. This is stored in a separate byte array, while the original input array is overwritten with null values ​​after processing to minimise security risks.

The signature is verified by initialising the Signature instance again, this time with the public key. The received data is passed to signature verification in byte form, ensuring that it matches the originally signed data. If the verification is successful, it means that the data has remained unchanged and comes from the specified source. On the other hand, a failure of verification indicates that either the data or the signature was manipulated or that an incorrect public key was used for verification.

The following source code shows an example usage:

javaCopy

importjava.security.KeyPair;importjava.security.KeyPairGenerator;importjava.security.PrivateKey;importjava.security.PublicKey;importjava.security.Signature;importjava.util.Arrays;publicclassDigitalSignatureExample{publicstaticvoidmain(String[]args)throwsException{KeyPairGeneratorkeyPairGenerator=KeyPairGenerator.getInstance("RSA");keyPairGenerator.initialize(2048);KeyPairkeyPair=keyPairGenerator.generateKeyPair();PublicKeypublicKey=keyPair.getPublic();PrivateKeyprivateKey=keyPair.getPrivate();char[]message={'H','e','l','l','o',' ','J','C','A'};byte[]signature=signData(message,privateKey);booleanisValid=verifyData(message,signature,publicKey);System.out.println("Signature valid: "+isValid);Arrays.fill(message,'\0');}privatestaticbyte[]signData(char[]input,PrivateKeykey)throwsException{Signaturesignature=Signature.getInstance("SHA256withRSA");signature.initSign(key);byte[]inputBytes=newbyte[input.length];for(inti=0;i<input.length;i++){inputBytes[i]=(byte)input[i];}signature.update(inputBytes);byte[]signedBytes=signature.sign();Arrays.fill(inputBytes,(byte)0);returnsignedBytes;}privatestaticbooleanverifyData(char[]input,byte[]signatureBytes,PublicKeykey)throwsException{Signaturesignature=Signature.getInstance("SHA256withRSA");signature.initVerify(key);byte[]inputBytes=newbyte[input.length];for(inti=0;i<input.length;i++){inputBytes[i]=(byte)input[i];}signature.update(inputBytes);booleanisValid=signature.verify(signatureBytes);Arrays.fill(inputBytes,(byte)0);returnisValid;}}

Now let’s combine symmetric and asymmetric encryption

Secure communication between a client and a server can be achieved by combiningasymmetric cryptography for key exchange andsymmetrical encryption for the actual message transmission. In this simulation, a hybrid encryption system is implemented that initially uses RSA to enable secure transmission of a session key, which is then used for encrypted communication using AES. This architecture is comparable to the handshake process inTLS (Transport Layer Security) , whereby asymmetric cryptography is only used for the initial key exchange. At the same time, the more efficient symmetric encryption secures the messages.

First, the server generates anRSA key pair , which is used to encrypt and decrypt the session key. The client provides the public key while the private key remains on the server. The client then creates onerandom AES session key , which is encrypted with and sent to the server’s RSA public key. Once received, the server decrypts the session key using its private key. Both parties now have the same symmetrical key used for subsequent encrypted communication.

The message transmission takes place using theAES algorithm in Galois/Counter Mode (GCM) , which, in addition to confidentiality, also ensures integrity protection through integrated authentication. Each message packet sent is given a random initialisation vector (IV) before encryption to ensure that identical messages do not result in identical ciphertexts.

If you remember the login process, you can use this procedure to secure the components transmitted over the network, such as hash values ​​and salt values. At this point, I must, of course, point out again that this is a presentation of the basic principles. For productive use, it is recommended to use existing implementations and protocols.

Happy Coding

Sven

]]>JavaSecure Coding PracticesSecurityUncategorizedhttps://svenruppert.com/posts/rethinking-java-streams-gatherer-for-more-control-and-parallelism/Wed, 02 Apr 2025 20:58:21 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/rethinking-java-streams-gatherer-for-more-control-and-parallelism/Since version 8, Java has introduced an elegant, functional approach to processing data sets with the Streams API. The terminal operation collect(…) represents the bridge from the stream to a targeted aggregation - in the form of lists, maps, strings or more complex data structures. Until Java 20 the processing was done Collector-Instances were regulated, which internally consisted of a supplier, an accumulator, a combiner and optionally a finisher. This model works well for simple accumulations but has noticeable limitations, particularly for complex, stateful, or conditional aggregations.<![CDATA[

Since version 8, Java has introduced an elegant, functional approach to processing data sets with the Streams API. The terminal operation collect(…) represents the bridge from the stream to a targeted aggregation - in the form of lists, maps, strings or more complex data structures. Until Java 20 the processing was done Collector-Instances were regulated, which internally consisted of a supplier, an accumulator, a combiner and optionally a finisher. This model works well for simple accumulations but has noticeable limitations, particularly for complex, stateful, or conditional aggregations.

1. The semantics of gatherers
2. Why Gatherers are more than just a “better collector”
3. Integration with Streams API
   1. Sequentieller Gatherer
      1. Initialisation: The state of the gatherer
      2. Accumulation: Processing the input elements
      3. Emission: The control over the output
      4. Signature and purpose of the finisher
      5. Concrete example: chunking with remainder
      6. Interaction with accumulation
      7. No return – but effect through push()
   2. Parallel Gatherer
      1. Initializer – The creation of the accumulator
      2. Integrator – The processing of elements
      3. Combiner – The combination of partial accumulators
      4. Finisher – The transformation of the accumulator into the final result
      5. Interaction in parallel gatherers
4. An example implementation
   1. Initialiser:
   2. Integrator:
   3. Combiner:
      1. entries state1 insert
      2. entries state2 insert
   4. Finisher:

Java 21 was the new interface java.util.stream. The gatherer was introduced, significantly expanding the semantics and control over the accumulation process. A Collector passively collects data, acts as a Gatherer, and actively responds to the incoming elements, which is comparable to a specialised transducer in functional programming languages. Gatherers are particularly useful where procedural or stateful aggregation is necessary, and they also allow element insertion, filtering, skipping, and explicit termination of the gathering process - all within the framework of a functional composable architecture.

The semantics of gatherers

A Gatherer<T, R> describes the transformation of one Stream into a result of type R under close control over the accumulation process. In contrast to Collector, which is in a sense a container for aggregation logic, the gatherer allows rule-based, state-dependent processing of inputs - including the ability to skip elements (Drop), to insert additionally (Inject) or to end processing early (FinishEarly).

To make this possible, a gatherer is based on the idea of ​​aSink , which is called in the context of the stream processor. This sink receives every input element, can react to it and thus actively influences the flow of processing. The actual processing is done via a so-calledAdapter Factory which manages the transitions between the aggregation states.

Why Gatherers are more than just a “better collector”

While the conventional Collector serves primarily as a final accumulation tool - i.e. to transfer the elements contained in the stream into a target structure such as a list, a map or an aggregation - it works Gatherer conceptually far beyond this role. It is not just an optimised or syntactically varied form of Collectors, but rather an independent mechanism that opens up new expressions for stream processing, both semantically and functionally.

The central difference lies in the expanded scope for action during the transformation: This means explicitly that a Gatherer can accumulate not only elements but also new, previously non-existent elements that can be fed into the data stream. This opens up the possibility, for example, of introducing initialisation values ​​at the beginning of a stream or placing control characters such as headers and footers specifically at the beginning or end - without artificially expanding the original data stream.

This creative freedom becomes particularly clear when dealing with conditions. Where a Collector usually operated with a simple accumulator, the state of which leads to a final result, can be a Gatherer work based on state – and allow this state to be influenced across several elements. This opens up new semantic horizons: For example, window operations can be formulated in which temporal or sequential logic is applied - such as aggregating data up to an inevitable “end” marker, or combining groups of elements that can only be identified by a particular order or content structure.

Even complex decision structures, such as those required in multi-stage parsing processes or when traversing decision trees, can be achieved using stateful ones that Gatherer implements elegantly and declaratively. The interface remains in the spirit of functional programming: transformation and aggregation can still be described separately, but the Gatherer ensures their connection in a way that was previously only possible through imperative or difficult-to-maintain stream hacks.

Another advantage is the controlled influence of past elements on current behavior. This is how one can Gatherer For example, making the decision to discard an element because a previous element set a certain context. This context sensitivity capability is particularly relevant in situations where data streams are structurally “not clean” – such as log files, inconsistent data exports, or natural language analysis.

A Gatherer is not just “better Collector", but a fundamental new tool for data stream-based modeling of complex transformation logic. It opens up a design space in which state, transformation, context and accumulation can interact in a way that was previously only possible with considerable effort - or outside of the stream model. Anyone who has ever worked with stateful Gathererconstructions will notice how much this expands the expressiveness of functional stream architectures.

A concrete example: grouping with filter logic

Let’s imagine that we want to collect from a stream of strings only those elements that have a particular property and then group them - for example, all words longer than 5 characters, grouped by their first letter. This requirement can be met with a Collector formulate, but requires a combination of preprocessing (e.g. filter(…)) and downstream grouping. With a Gatherer On the other hand, this combined process can be represented elegantly, comprehensively and in one step:

xmlCopy

Gatherer<String,?,Map<Character,List<String>>> gatherer = Gatherer.ofSequential( () -> new HashMap<Character,List<String>>(), (map, element, downstream) -> { if (element.length() > 5) { char key = element.charAt(0); map.computeIfAbsent(key,k -> new ArrayList<>()).add(element); } return true; } );

In this example, a decision is made for each element as to whether it will be included in the result. The logic is embedded directly into the Gatherer. The return value accurate signals that processing should continue. You would, at this point, instead of false return, and the stream would end prematurely - a behaviour that is not possible with conventional Collectors is not reachable like that.

Integration with Streams API

The interface Gatherer<T, A, R> explicitly distinguishes betweenmore sequential andparelleler processing. The central distinction arises from the factory methods:

Gatherer.ofSequential(…) // Can only be used sequentially

Gatherer.ofConcurrent(…) // Suitable for parallel streams

A gatherer who comes withofConcurrent(…) may be used in parallel streams, but must meet certain requirements: it must be thread-safe or rely on thread-isolated accumulators. This is similar to the logic of parallel collectors, where internal state management allows different elements to be processed simultaneously in independent threads.

Sequentieller Gatherer

Especially atsequential processing —i.e., if there is no parallelisation—the Gatherer develops its full expressiveness while remaining simple, type-safe, and deterministic.

The functionality of a sequential gatherer can be divided into three main phases:initialisation ,accumulation andEmission. Each of these phases is described in detail below, with particular attention to the unique features of sequential processing.

Initialisation: The state of the gatherer

Each gatherer has an internal state that is recreated per stream execution. This condition is about one Supplier defined, where S represents the type of condition. In sequential processing, this state is reserved exclusively for a single thread; therefore, nothread safety requirements exist. This means that simple objects like ArrayList, StringBuilder, counter arrays, and custom records can be used without problems.

A typical condition could e.g. B. look like this:

record ChunkState(List buffer, int chunkSize) {}

The associated supplier:

() - > new ChunkState<>(new ArrayList<>(chunkSize), chunkSize)

The state lives for the entire stream run and serves as context for all further processing steps. Initialisation lays the foundation for state-based logic, such as buffering, counting, aggregation or tracking previous elements.

Accumulation: Processing the input elements

The accumulation function is the heart of every gatherer. It is called for each element of the input stream. The signature of this function is:

(state, input, downstream) - > { … }

This is where the actual logic happens: The input element is brought into the state, and - depending on the current state - a decision can be made as to whether (and if necessary, how many) output values ​​are produced. The decision as to whether an item is passed downstream rests entirely with the gatherer.

Example: Every third element of a stream should be emitted.

javaCopy

Gatherer<String,?,String>everyThird(){returnGatherer.ofSequential(()->newint[]{0},(state,element,downstream)->{state[0]++;if(state[0]%3==0){downstream.push(element);}});}

In contrast to classic filter or map operations, this logic isconditional andimperative : The gatherer remembers how many elements have already been processed and only emits a result for every third. The accumulation logic is, therefore, comparable to that of the accept () Method of a specially defined consumer, but supplemented by downstream control.

Since there is no threading in sequential processing, all operations can be performed directly without synchronisation. The state can be arbitrarily complex and dynamic as long as it is updated correctly within the stream.

Emission: The control over the output

Elements are output via the Sink-Object provided to the Gatherer upon each accumulation. With his method, push(R element) elements can be passed downstream in a targeted and controlled manner. Instead of map or flatMap, where each input leads to one or more outputsautomatically transformed, the gatherer decides himself,if ,at andwas he emits.

For example, a gatherer can:

• push individual output values,
• push multiple values ​​at once (e.g. with chunking or tokenisation),
• completely suppress the emission (e.g. under preconditions),
• generate values ​​with a delay (e.g., at the stream’s end or after accumulation thresholds).

Example: Combining three consecutive elements into a string:

xmlCopy

Gatherer<String,?,String> triplets() { return Gatherer.ofSequential( () -> new ArrayList<String>(3), (state, element, downstream) -> { state.add(element); if (state.size() == 3) { downstream.push(String.join("-", state)); state.clear(); } });}

The emission here only occurs when three elements have been collected. These are merged, pushed and the state is then emptied.

An often overlooked but essential component of a sequential gatherer is theFinisher – i.e. the closing function after the last input element. This phase is crucial because often during regular accumulationItems retained in state which will only be done at a later date or evenno longer through regular accumulation can be emitted. The finisher ensures that such remaining elements or aggregated partial results are not lost but are correctly transferred downstream.

Signature and purpose of the finisher

The closing function has the signature:

BiConsumer <State, Sink>

she willafter all input values ​​have been processed called by the stream framework – exactly once. In this function, the final state can be accessed and, based on this state, a decision can be made as to whether (additional) output values ​​should be created.

The finisher is particularly suitable for:

• Partially filled buffers , for example in chunking operations when the last block does not reach full size,
• Final aggregations , e.g. B. in averaging, summation, hash calculation or protocol completion,
• Finalisation of state machines , e.g. B. if an open state still needs to be completed,
• Cleaning or logging , e.g. B. statistical outputs or final indicators.

Concrete example: chunking with remainder

Let’s look again at the example of a gatherer that groups elements into groups of three. Without finishers, if the number of elements is odd, the last one or two values ​​would be lost:

xmlCopy

Gatherer<String,?,String> triplets() { return Gatherer.ofSequential( () -> new ArrayList<String>(3), (state, element, downstream) -> { state.add(element); if (state.size() == 3) { downstream.push(String.join("-", state)); state.clear(); } }, (state, downstream) -> { if (!state.isEmpty()) { downstream.push(String.join("-", state)); } } );}

In the closing function (finish), it is explicitly checked whether there are still elements in the state—i.e., whether the buffer is incomplete. These residual values ​​are then combined into an aggregate and pushed.

Without the finisher there would be the gathererfunctionally incomplete : For input sets that are not a multiple of three, the last chunk would simply be discarded - a classic off-by-one error.

Interaction with accumulation

The finisher issemantically separated from the accumulation logic, but accesses the same state. This means that, depending on the specific application, it can use the same auxiliary functions or serialisation routines as the accumulation itself. In practice, it is advisable to define auxiliary methods for common logic such as “combine and empty the list” in order to avoid redundancy.

No return – but effect through push()

The finishergives no return value back, but - like the accumulation function - works via what is provided Sink. So it doesn’t find any return semantics, instead a controlled completion of processing push() views.

The finisher of a sequential gatherer is thebinding conclusion of the processing model. He guarantees thatall information remaining in the state is processed and, if necessary, emitted. Especially in data stream-based applications where incomplete blocks, open ends, or residual states are typical, the finisher is essential to avoid data loss and ensure semantic correctness. Therefore, the finisher has a clean gatherer designthat is not optional but rather an integral part of a well-defined stream processing step.

A sequential gatherer combines:

• the state handling of an aggregator,
• the control flow options of a parser,
• the expressiveness of an imperative processor,
• and the clarity of functional APIs.

By foregoing parallelisation logic and concurrency, the sequential variant allows a gatherer to be developed with minimal overhead and maximum expressiveness - a tool that combines both the flexibility of imperative logic and the composition of functional programming.

Parallel Gatherer

A parallel gatherer is forparallel Data processing pipelines are responsible for the four phases initialiser, integrator, combiner and finisher can be explicitly separated and controlled from each other.

Initializer – The creation of the accumulator

The method initialiser defines how a new accumulator (internal state) is created. This is the first step in processing each substream in sequential and parallel pipelines.

The signature is also:Supplier initializer();

In parallel processing, this initialiser is called several times - once per substream, i.e. per thread that takes over a split of the data. This ensures that no synchronisation within the accumulator is necessary: ​​each thread operates in isolation with its own state.

Integrator – The processing of elements

The integrator is a central function for inserting stream elements into the accumulator. It is one BiConsumer<A, T>, i.e. a function for each element T the Accumulator A changed accordingly.

The signature reads:BiConsumer <A, ? super T> integrator();

In parallel streams, this integrator is also applied to partial accumulators. What is important here is that this function may only change the accumulator locally and may not influence any global states.

Combiner – The combination of partial accumulators

The combiner’s task is to combine several independently processed accumulators into one overall accumulator. This phase is only relevant in parallel stream executions. The combiner receives two partial results—typically from two threads—and has to combine them into a common result.

The signature is:BinaryOperator combiner();

The correct implementation of the combiner is essential for the correctness of the parallel execution. It must be associative. This is the only way the JVM can freely distribute the operation across multiple threads.

Finisher – The transformation of the accumulator into the final result

The finisherfunction transforms the accumulator A into the desired result R. While A is used internally to work efficiently during aggregation R the actual result of the entire operation - such as an immutable collection, a merged string, an optional, a report object, etc.

The signature reads:Function <A, R> finisher();

Unlike the integrator and combiner, the finisher becomes accurateonce called, at the end of the entire processing chain. It therefore serves as a bridge between the internal aggregation mechanism and external representation.

Interaction in parallel gatherers

In a parallel stream with gatherer-based collector, the following happens:

1. An initialiser is called for each partial stream (split) to create a local accumulator.
2. The integrator processes all elements in the respective substream one after the other and modifies the local accumulator.
3. The combiner phase is called as soon as two partial accumulators need to be combined. This happens either through ForkJoin merging or in the final reduction step.
4. After all partial accumulators have been combined, the finisher is called to calculate the final result.

This explicit separation and the ability to control each phase make Gatherer a powerful tool for complex, stateful, or stateless aggregations—especially when performance through parallelisation is crucial or custom aggregation logic is required.

An example implementation

Let’s first define the gatherer in general and put it in a stream.

xmlCopy

Gatherer<Integer,?,ConcurrentMap<Integer,List<Integer>>> concurrentGatherer = Gatherer.of( initializer, integrator, combiner, finisher );IntStream.rangeClosed(0, END_INCLUSIVE) .boxed() .parallel() .gather(concurrentGatherer) .forEach(e -> System.out.println("e.size() = " + e.size()));

Now we define the respective subcomponents:

Initialiser:

var initializer = (Supplier <ConcurrentMap<Integer, List») ConcurrentHashMap::new;

Integrator:

xmlCopy

var integrator = new Gatherer.Integrator< ConcurrentMap<Integer,List<Integer>>, Integer, ConcurrentMap<Integer,List<Integer>>>() { @Override public boolean integrate(ConcurrentMap<Integer,List<Integer>> state, Integer element, Gatherer.Downstream< ? super ConcurrentMap<Integer,List<Integer>>> downstream) { if (element > END_INCLUSIVE) return false; //processing can be interrupted int blockStart = (element / 100) * 100; state .computeIfAbsent(blockStart, k -> Collections.synchronizedList(new ArrayList<>())) .add(element); return true; }};

The integrator is responsible for processing individual stream elements (here: Integer) and their insertion into a shared state (ConcurrentMap). The element is sorted according to a specific grouping criterion: all elements that are in the same block of 100 (e.g. 0–99, 100–199, …), are entered in the same list.

There is a special feature in this implementation:

if (element > END_INCLUSIVE) return false;

This condition serves asAbort signal : as soon as an element is above a specified limit (END_INCLUSIVE), processing is completed by returning false canceled. This is a special feature of the Gatherer-Model: The return value false signals that no further elements should be processed - a typeearly termination.

javaCopy

state.computeIfAbsent(blockStart,k->Collections.synchronizedList(newArrayList<>())).add(element);

This line does the actual grouping: If under the key blockStart, if no list already exists, a new, synchronised one will be created. ArrayList created.

The current item is then added to this list.

By using Collections.synchronizedList(…) becomeseven within a parallel gatherer context ensures that list accesses are thread-safe - even though the ConcurrentMap itself is only responsible for map access, not for the values ​​it contains.

The integrator therefore defines the following processing semantics:

• Elements are grouped by blocks of 100 (0–99, 100–199, etc.).
• The assignment is done via a ConcurrentMap, where each block contains a list.
• The lists themselves are synchronised to allow concurrency within the list operations.
• By returning false can processingended early become – e.g. B. when a limit value is reached.

Combiner:

xmlCopy

var combiner = new BinaryOperator<ConcurrentMap<Integer,List<Integer>>>() { @Override public ConcurrentMap<Integer,List<Integer>> apply(ConcurrentMap<Integer,List<Integer>> state1, ConcurrentMap<Integer,List<Integer>> state2) { var mergedMap = new ConcurrentHashMap<Integer,List<Integer>>(); // fill in state1 state1.forEach((key, value) -> mergedMap.merge(key, value, (v1, v2) -> { v1.addAll(v2); return v1; }) ); // fill in state 2 state2.forEach((key, value) -> mergedMap.merge(key, value, (v1, v2) -> { v1.addAll(v2); return v1; }) ); return mergedMap; }

First, a new empty, thread-safe map is prepared to contain all entries state1 and state2 should be inserted. This new map is deliberately fresh because neither state1 still state2 should be changed - this protects against unwanted side effects and makes the function workreferentially transparent.

var mergedMap = new ConcurrentHashMap <Integer, List>();

entries state1 insert

javaCopy

state1.forEach((k,v)->{mergedMap.computeIfAbsent(k,k1->newArrayList<>()).addAll(v);});

This methodcomputeIfAbsent checks whether in the target map mergedMap already an entry for the key k exists. If this is not the case, the lambda is used k1 -> new ArrayList<>() a new entry is created and inserted. The method guarantees thatan existing, modifiable list is returned afterwards - regardless of whether it was just created or already existed.

The method addAll(…) hangs all elements of the list v out of state1 to the corresponding list in mergedMap to. This expands the aggregate state for this key in the target map.

entries state2 insert

The same process is then repeated for state2 repeated:

javaCopy

state2.forEach((key,value)->mergedMap.merge(key,value,(v1,v2)->{v1.addAll(v2);returnv1;}));

Every entry is made here state2 in the mergedMap transmitted. If the key does not yet exist, the value (value, one List) taken directly. If the key is already in mergedMap, it exists by merge(…) using a custom merge strategy: the list contents of both maps are merged v1.addAll(v2) combined. What is important here is that v1 is the existing list, and v2 is the newly added one.

In the end, the newly created, combined map is returned—it represents the complete aggregate state, which contains the contents of both partial states.

Finisher:

xmlCopy

var finisher = new BiConsumer< ConcurrentMap<Integer,List<Integer>>, Gatherer.Downstream<? super ConcurrentMap<Integer,List<Integer>>>>() { @Override public void accept( ConcurrentMap<Integer,List<Integer>> state, Gatherer.Downstream<? super ConcurrentMap<Integer,List<Integer>>> downstream) { downstream.push(state); }};

This implementation of the finisher is minimalistic but functionally correct: it takes the final state (state) of the accumulator - one ConcurrentMap<Integer, List> – and hands it directly to himdownstream , i.e. the next stage of the processing chain in the stream.

The attributestate is the aggregated result of the previous steps (initialiser, integrator, combiner). In this case it is a map that maps blocks of 100 (Integers) to a list of the associated values ​​(List).

The attributedownstream is a push receiver that consumes the end result. It abstracts the next processing stage, such as a downstream map, flatMap, or terminal collection process.

The method push(…) of the downstream object explicitly forwards the finished result to the next processing stage. This is fundamentally different from conventional collector Concepts, where the end result is simply returned.

This type of handover makes it possible, in particular, in onewithin Gatherer defined, stateful context to deliver multiple or even conditional results – for example:

• Streaming of intermediate results (e.g. after a specific batch)
• Quitting early after the first valid result
• Multiple edition during partitioning

In this specific case, however,precisely one result was passed on—the fully aggregated map. This classic “push-forward finisher” determines the condition as a resultemitted.

We have now examined the Gatherer in detail and pointed out the differences in sequential and parallel processing. So, everything should be together for your first experiments.

Happy Coding

Sven

]]>ConcurrencyJavaStreamshttps://svenruppert.com/posts/from-java-8-to-24-the-evolution-of-the-streams-api/Sat, 29 Mar 2025 13:08:23 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/from-java-8-to-24-the-evolution-of-the-streams-api/The introduction of the Stream API in Java marked a crucial step in the development of functional programming paradigms within the language. With Java 24, stream processing has been further consolidated and is now a central tool for declarative data processing Stream not about an alternative form of Collection, but rather an abstract concept that describes a potentially infinite sequence of data that is transformed and consumed through a pipeline of operations. While a Collection is a data structure that stores data, Stream is a carrier of a computing model: It does not store any data but instead allows the description of data flows.<![CDATA[

The introduction of the Stream API in Java marked a crucial step in the development of functional programming paradigms within the language. With Java 24, stream processing has been further consolidated and is now a central tool for declarative data processingStream not about an alternative form of Collection, but rather an abstract concept that describes a potentially infinite sequence of data that is transformed and consumed through a pipeline of operations. While a Collection is a data structure that stores data, Stream is a carrier of a computing model: It does not store any data but instead allows the description of data flows.

1. Getting started with streams in Java
2. The power of composition
3. Effects on the Classic Design Patterns in Java
4. An example from classic Java to the use of streams
   1. Example in classic Java without streams
   2. Example in Java using Streams API
5. Changes to the Streams API from Java 8 to 21 - an overview
6. A concrete example of flatMap and mapMulti
   1. Conceptual difference
   2. Example with flatMap
7. What’s new with Java 24 - Gatherer
8. Practical examples with the Gatherer
   1. windowFixed
   2. windowSliding
   3. possible optimizations

Each stream processing can be divided into three logically separate phases. The starting point is always a source - one Collection, an array, an I/O channel or a created structure like Stream.generate(…) be. The source is followed by a number ofIntermediate Operations , which transform, filter or sort the Stream. These operations arelazy , d. h. they are not executed immediately but are merely registered. Only through a finalTerminal Operation , such as forEach, collect or reduce, execution is triggered and the entire pipeline is concretized. TheLaziness principle allows the JVM to optimize, merge, or even eliminate operations without impacting the final result.

A fundamental feature of stream architecture is its uniqueness. A stream can be consumed exactly once. Once a terminal operation has been performed, the Stream is exhausted. This has profound implications for the design of stream-based algorithms, as one must carefully decide when and where to evaluate a pipeline. Compared to reusable data containers like lists, this requires a shift towards fluent, targeted processing.

The structure of a stream pipeline follows a well-defined structure. It begins with the definition of the data source, is completed by a sequence of intermediate operations and ends with a final consumption action. Under the hood, the JVM detects and analyzes this structure and performs a variety of optimizations. For example, the JVM can do operations like map and filter combine (Operation Fusion), activate parallel execution on suitable sources or avoid redundant calculations. These optimization options are only possible due to the declarative nature of streams and underline the paradigmatic difference to imperative code with classic loops.

In Java, the Stream API remains a key element for modern, expressive, and concurrent computing. However, understanding them requires a deep awareness of the differences from the conventional collection hierarchy - particularly in terms of laziness, single use, and JVM-powered optimizations that make streams a powerful tool in the Java developer’s repertoire.

Getting started with streams in Java

Getting started with the Stream API in Java requires a basic understanding of functional programming concepts and the way Java integrates these concepts within an object-oriented language. Streams make it possible to describe data flows declaratively, with the focus not on theHow but on theWhere the data processing lies. This shift from imperative to declarative thinking forms the foundation for a modern, concise and, at the same time, expressive programming model.

A typical entry point into stream processing is the conversion of existing data structures, such as List- or Setinstances, into a stream. This is done using the method stream(), by most Collection implementations is provided. Alternatively, methods such as: Stream.of(…), Arrays.stream(…) or IntStream.range(…) can be used to create stream instances. Regardless of the source, this always creates a pipeline that is potentially evaluated with a delay.

After a stream is initialized, intermediate operations such as filter, map, or sorted are applied to transform or refine the data flow. These operations are purely descriptive and do not modify the original data source or trigger any calculation. Instead, you build a kind of recipe that is later executed through a terminal operation.

As mentioned before, the pipeline is completed by a terminal operation that triggers execution and delivers the result material. Only at this point is the pipeline evaluated, with all previously defined steps being applied to the data in one pass. This is a key difference from traditional processing using loops or iterators, where each processing step is executed immediately.

Especially at the beginning, it is helpful to experiment with simple examples, such as filtering character strings or transforming integers. This allows an intuitive understanding of the data flow, the chaining of operations and the underlying execution logic. The advantage of streams quickly becomes apparent: they promote a concise, readable and at the same time efficient expression for data-driven operations, without unnecessarily complicating the structure of the underlying code.

The power of composition

The Java Stream API’s strength lies in its ability to process data declaratively and, above all, in its ability to form complex processing steps into a compositional unit. Streams allow transformation and filter logic to be defined modularly and combined elegantly, making even complex data flows comprehensible and reusable. The underlying idea is to understand arithmetic operations as functional units that can be fluidly combined into a processing chain.

From a software architecture perspective, stream composition opens an elegant way to modularize processing steps. For example, methods can be defined as those that encapsulate a specific stream transformation and can be embedded as a building block in a more extensive pipeline. By using higher-order functions, for example, by returning a Function<T, R>-Object, a repertoire of reusable processing modules is created that can be put together dynamically. This not only promotes readability, but also testability and maintainability of the code.

Another advantage is the clear separation of structure and behaviour. While imperative code typically mixes loop logic with specialized logic, stream composition allows a decoupled view: the type of data flow (e.g. sequential or parallel) is orthogonal to the question of what happens to the data in terms of content. This separation makes it easier to reuse or specifically expand existing pipelines in different contexts without fundamentally changing their structure.

Last but not least, the combination of different stream types expresses this compositional ability. So primitive streams (IntStream, LongStream, DoubleStream) with object streams via conversion methods like boxed() or mapToInt() be connected. The same can be said about flatMap, which realizes the processing of nested data structures, whereby a stream of containers is created into a flat data stream that can be processed further seamlessly.

Therefore, the ability to compose streams is more than just a syntactic convenience. It represents a fundamental paradigm shift in the way data flows are designed, structured, and executed in Java. Anyone who masters this approach can convert even complex application logic into clearly structured, functional modules—a gain in terms of both maintainability and expressiveness.

Effects on the Classic Design Patterns in Java

The integration of the Stream API into Java has profound implications for the use and necessity of classic design patterns. Many patterns that emerged in object-oriented development emerged in response to the lack of functional means of expression. However, with the emergence of streams and their close integration with functional concepts such as lambdas and higher-order functions, the role of these patterns is fundamentally changing. This is particularly noticeable with patterns that were initially used to control the iteration, transformation or filtering of data structures.

A prominent example of this is theIterator Pattern , which has been primarily made obsolete by streams. While the Iterator Pattern in classic Java was considered an idiomatic approach to processing collections sequentially, the Stream API completely encapsulates this responsibility. Access to the elements is implicit and declarative; the stream configuration controls the order and traversal behaviour. Manual control over iteration, as provided by the iterator pattern, is thus replaced by a more expressive and, at the same time, less error-prone abstraction.

That tooStrategy Pattern undergoes a transformation through streams. Strategies that were previously coded as separate classes or interfaces with concrete implementations can now be elegantly implemented via lambdas and method-based composition within stream pipelines. Filtering or mapping strategies that were previously modelled through object-oriented inheritance hierarchies can now be defined inline and dynamically combined - a change that not only simplifies the source code but also increases its flexibility.

TheDecorator Pattern , traditionally used to extend functionality through nested object structures dynamically, finds a modern equivalent in the stream world in the chaining of intermediate operations. Each operation transforms the Stream and adds another processing layer - not through inheritance or object composition, but through functional pipeline elements. The resulting structure is more compact and allows for a much more dynamic configuration at runtime.

Furthermore, stream architecture sheds new light on theTemplate Method Pattern. While this was initially used to define the structure of an algorithm and implement varying steps in subclasses, the Stream API allows such “algorithms” to be defined via methodically combined functions. The sequence of stream operations specifies the fixed processing steps, while passed functions specify concrete steps. This makes the behaviour more modular and decoupled from static inheritance.

Finally, streams also change the understanding of patterns likeChain of Responsibility orPipeline. These patterns are designed to model flexible processing sequences in which each element acts optionally and can pass responsibility. Stream pipelines implement this principle at a functional level, with each intermediate operation corresponding to a “processing unit”. The big difference is that there is no need to chain objects together manually; instead, the processing chain results from the fluent syntax of the API itself.

Overall, streams in Java lead to a functional re-contextualization of classic design patterns. This does not make many patterns superfluous but instead transforms them in their implementation. They now appear less rigid class structures and more as dynamic, composable units that seamlessly fit into fluent APIs. For the informed developer, this not only opens up new possibilities for expression but also a reflective understanding of when a pattern in the classic sense is still necessary. But I will report on this in a separate post.

An example from classic Java to the use of streams

A classic data processing algorithm in Java extracts, transforms, and aggregates information from a list of complex objects. Let’s take a list of as an example Person-Objects, each containing name, age and place of residence. The goal is to extract all the names of adults, sort them alphabetically and return them as a string separated by commas. The classic imperative-object-oriented implementation of this requirement in Java typically takes place via explicit loops, temporary lists and manual control structures.

You usually start by creating a new results list in such an implementation. The original data is then iterated over, in one if-Condition it is checked whether the age of the person in question is over 17. If so, the person’s name is added to the results list. Once the iteration is complete, the list is sorted alphabetically by explicitly calling the sort method. Finally, the list is created using a loop or by using a StringBuilder converted into a comma-separated string. This approach works correctly but requires a lot of intermediate steps and quickly leads to redundant or error-prone logic - for example, when sorting or formatting the result.

With the introduction of the Stream API, the same algorithm can be expressed in a much more concise and declarative manner. The entire data flow – from filtering to transformation to aggregation – can be modeled in a single expression unit. The Stream is derived from the list, then filters filter-Operation to remove adults. A subsequent one map-Transformation extracts the names. The sorting is done by sorted realized, and the final aggregation takes place via Collectors.joining(","). The entire algorithm can, therefore, be expressed in a fluent, clearly structured pipeline, the sub-steps of which are semantically self-explanatory through their method names.

The differences between the two approaches are both syntactic and conceptual. While the classic solution relies heavily on imperative thinking and controls each processing element manually, the stream approach allows declarative modeling of the “what” and leaves the “how” to be executed by the runtime environment. This level of abstraction promotes readability and maintainability of the code, as the developer can concentrate on the technical logic without being distracted by control flow constructs.

At the same time, it can be seen that streams also enable semantic compression. An algorithm that previously required several dozen lines is often reduced to a few clearly structured function calls. However, this compactness does not come at the expense of transparency - on the contrary, the descriptive method names and the fluent structure make the data flow explicitly understandable. The stream variant, therefore, not only appears more modern but also conceptually closer to the problem.

It can be said that the stream-based implementation integrates functional principles into the Java world without sacrificing type safety or object orientation. It makes it possible to formulate classic algorithms with a new level of abstraction that not only simplifies the code, but also expresses its intent more clearly.

But let’s look at it in the source code: Let’s assume that we have the following structures available for both implementations.

xmlCopy

record Person(String name, int age, String city) {}private static final List<Person> PEOPLE = List.of(   new Person("Alice", 23, "Berlin"),   new Person("Bob", 16, "Hamburg"),   new Person("Clara", 19, "München"),   new Person("David", 17, "Köln"));

Example in classic Java without streams

xmlCopy

List<String> adultNames = new ArrayList<>(); for (Person p : PEOPLE) { if (p.age >= 18) { adultNames.add(p.name); } } Collections.sort(adultNames); StringBuilder result = new StringBuilder(); for (int i = 0; i< adultNames.size();i++){result.append(adultNames.get(i));if(i<adultNames.size()-1){result.append(",");}}System.out.println("Ergebnis:"+result);}

In the classic variant, the data flow is fragmented and distributed over several steps: filtering, collecting, sorting and formatting take place in separate sections, sometimes using temporary structures. This leads to increased complexity and potential sources of errors - for example when formatting the output string correctly.

Example in Java using Streams API

javaCopy

publicstaticvoidmain(String[]args){Stringresult=PEOPLE.stream().filter(p->p.age()>=18).map(Person::name).sorted().collect(Collectors.joining(", "));System.out.println("Ergebnis: "+result);}

The stream variant, on the other hand, expresses the entire data processing process in a single pipeline. Each processing step is clearly named, the transformation takes place fluidly, and the result is immediately visible. Particularly noteworthy is the better onereadability , theReduction of side effects and theExtensibility – additional processing steps can be added by simply inserting additional operations into the pipeline without fundamentally changing the structure.

Changes to the Streams API from Java 8 to 21 - an overview

Since its introduction in Java 8, the Stream API has fundamentally changed the way data is processed in Java. With the aim of enabling declarative, functionally inspired data flows, she established a new programming paradigm within the object-oriented language. The initial version was already remarkably expressive: it offered a clear separation between data source, transformation and terminal operation, supported lazy evaluation and could be executed both sequentially and in parallel. In particular the integration with lambdas, method references and the java.util.functionlibrary enabled a fluid, type-safe style of data processing that was previously only possible via external libraries or explicit iterators.

With the subsequent versions of the JDK, the Stream API was not fundamentally redesigned, but was continually expanded, refined and stabilized. Java 9 brought with it the first significant enhancements, such as the methods causeWhile, dropWhile and iterate with predicates that made it possible to control stream pipelines even more precisely and terminate them early. These extensions closed a semantic gap in the original API and, in particular, improved the ability to model more complex data flows. Also the introduction of Optional. Stream () was a notable move as it provided an elegant bridge between the types Optional and Stream and thus further increased the ability to compose in a functional style.

In Java 10 to 14, priority was given to accompanying language features such as our introduced, which did not directly change the Stream API itself, but improved its readability and applicability in many contexts. It was only with Java 16 and the consistent spread of records as compact data containers that stream processing became more expressive again, as it could now be combined with structured but immutable data models - a clear advantage for parallel or deterministic processing scenarios.

Java 17, as a long-term support version, consolidated the API through additional optimizations in the backend and further improved integration with pattern matching and modern sealed-Class hierarchy. These structural improvements did not introduce new methods in the Stream class itself, but enabled more precise handling of heterogeneous data flows and the formulation of domain-specific pipelines at a high level of abstraction.

It was only Java 21 that brought tangible expansions in the area of ​​functional data processing, especially in interaction Scoped Values, structured concurrency and the further developed ForkJoinPool implementation. While the Stream API received hardly any new methods formally, its applicability in the context of concurrent and memory-optimized architectures was significantly strengthened. The increased performance with parallel execution and the more efficient management of intermediate results also reflect a maturation of the API at the implementation level. With Java 21, streams can not only be written more elegantly but also executed more securely and predictably - especially in asynchronous or reactive processing environments.

In summary, the Stream API has undergone a clear maturation process from Java 8 to Java 21. While its conceptual core remained essentially constant, its semantic scope was clarified, its integration with modern linguistic means was deepened, and its efficiency was optimized on several levels.

A concrete example of flatMap and mapMulti

Conceptual difference

flatMap was already introduced with Java 8 and is based on the idea of ​​applying a function that creates a new one for each element of the original streamStream produced. These nested streams are then “flat” merged so that the resulting Stream has a flat structure. This pattern is compelling and allows elegant transformations of nested data structures - for example Stream<List> to Stream.

mapMulti(), introduced in Java 16, takes a different approach. Instead of creating nested streams, one is created hereConsumer based Interface used: The method receives one for each element of the Stream BiConsumer, through which the developer can directly deliver zero, one or more output values ​​to the downstream - without creating temporary data structures such as lists or streams. This avoids heap allocations, which plays a particularly relevant role in performance-critical scenarios.

In the following example, let’s assume that we have the following data structure.

xmlCopy

private static List<List<String>> DATA = List.of( List.of("a", "b"), List.of("c"), List.of(), List.of("d", "e"));

Example with flatMap

xmlCopy

List<String> result = DATA .stream() .flatMap(List::stream) .collect(Collectors.toList());

Here each inner list is converted into a stream and then “flattened”.

xmlCopy

List<String> result = DATA .stream() .<String>mapMulti(Iterable::forEach) .toList();

In this case, no new stream instance is created. Instead, it is about the forEach-Loop the content passed directly to the consumer.

flatMap stands more for declarative clarity and conceptual simplicity. The method represents a fundamental principle of functional programming and is therefore particularly suitable if the transformation is based on existing Stream-generating functions takes place. It’s great for teaching, readability, and functional reasoning, but does introduce some overhead from temporary streams.

mapMulti(), however, aims tooptimize away. It allows transformations to be expressed more efficiently by granting direct control over the output. This reduces heap allocations, allows better memory locality and is therefore preferable from a JVM optimization perspective if performance is a priority. However, the semantic expression is more complex: the user must explicitly say so Consumer-Logic work, which can limit readability and comprehensibility for functionally less experienced developers.

You can say that flatMap, the idiomatic, functional way, remains, while mapMulti() represents a targeted tool for high-performance data processing. Their coexistence within the Stream API demonstrates Java’s ambition to combine both expressiveness and efficiency within the same paradigm - a balancing act between declarative elegance and system-level control.

What’s new with Java 24 - Gatherer

With Java 24, the Stream API has been expanded to include a new concept calledGatherer company. This addition addresses a long-standing gap in stream processing: the ability to perform custom aggregations across multiple elements in a controlled, stateful manner - but not at the end of the pipeline, as is the case with Collector-Instances is the case, butwhile of the stream process itself, as an integral part of the transformation. Gatherers represent a new class of intermediate operations that were specifically designed for advanced data flow logic.

The fundamental motivation for introducing gatherers is the limitation of the previous API in dealing with compound, multi-step, or stateful operations, particularly in the case of transformations that do not just produce a single element per input, but require sequencing, delaying, or grouping across multiple input elements. Previous tools like map, flatMap or peek all operate either statelessly or with limited context. For more complex cases, specialized iterator implementations or external libraries had to be used - which contradicts the goal of declarative data flow modelling in streams.

Gatherers solve this problem through a new processing model based on a controlled state. Similar to CollectorInstances define a set of functions that allow elements to be buffered, transformed and emitted - not at the end of the Stream, but as part of the ongoing data flow. So they combine the conceptual strengths of Collector and mapMulti, but extend these to include explicit support for temporary states and flexible emission strategies. There will be one Sink used to deliver any results downstream - even several per input element or even delayed.

The purpose of this expansion lies not only in expressivity but also in performance. Many use cases, such as sliding windows, temporal aggregations, sequence analysis or transformation logic with history, can be modelled precisely and efficiently with gatherers without sacrificing the declarative structure of the pipeline. This provides the developer with a tool that explicitly allows intermediate states and context dependency within stream processing but is type-safe, controlled and idiomatically embedded in the existing API.

This opens up new perspectives in development. Gatherers enable complex data flow modelling with a declarative character without falling back into imperative control logic. They significantly expand the semantic space of streams and thus represent a logical next step in the evolution of stream architecture - comparable to the introduction of Collector or mapMulti. Its introduction shows that Java integrates functional principles and develops them further in state models, process control and efficiency - without the compromises that often accompany system-level optimization.

Practical examples with the Gatherer

The introduction of gatherers not only created a new extension of stream processing but also provided a collection of predefined gatherers that address typical use cases that were previously difficult to model. These predefined gatherers combine declarative expressiveness with stateful transformation, enabling new forms of data stream processing - especially for sequential, grouped or sequentially correlated data flows. The central representatives of this new genre include windowFixed, windowSliding, fold and scan. Each of these gatherers brings specific behaviour that previously had to be implemented with considerable manual effort or even outside of the stream API.

windowFixed

The Gatherer windowFixed allows a stream to be divided into equally sized, non-overlapping windows. This is particularly relevant if data is to be further processed or aggregated in blocks. Suppose we want to split a list of integers into groups of five and calculate the sum of each group. With windowFixed This is very elegant:

xmlCopy

List<Integer> input = IntStream.rangeClosed(1, 15).boxed().toList();List<Integer> resultGatherer = input.stream() .gather(Gatherers.windowFixed(5)) .map(window -> window.stream().reduce(0, Integer::sum)) .toList(); // [15, 40, 65]System.out.println("resultGatherer = " + resultGatherer);

The alternative without a gatherer could look like this.

xmlCopy

List<Integer> result = new ArrayList<>();List<Integer> window = new ArrayList<>();for (int i : input) { window.add(i); if (window.size() == 5) { result.add(window.stream().reduce(0, Integer::sum)); window.clear(); }}System.out.println("result = " + result);

The difference is clear: The Gatherer variant is not only more compact, but also clearly separates data flow and logic, while the imperative solution works with side effects and state management.

windowSliding

A related but semantically more sophisticated case is given by windowSliding covered. These are sliding windows - each new element moves the window by one.

xmlCopy

List<Integer> input = List.of(1, 2, 3, 4, 5);List<Double> result = input.stream() .gather(Gatherers.windowSliding(3)) .map(window -> window.stream().mapToInt(Integer::intValue).average().orElse(0)) .toList(); // [2.0, 3.0, 4.0]

The given Java source code demonstrates an application des Gatherers windowSliding(n). This is used in this example to create sliding windows - so-called - over a given list of integersSliding Windows – to generate. The database in this case is an immutable list of integers:

List input = List.of(1, 2, 3, 4, 5);

The subsequent stream processing aims to calculate the arithmetic mean (average) of the elements contained over each sliding window of size three. The method windowSliding(3) causes the data stream to be internally split into overlapping windows, with each window containing three consecutive elements. For the list [1, 2, 3, 4, 5] This results in the following windows:

• [1, 2, 3]
• [2, 3, 4]
• [3, 4, 5]

Within the mapoperator will turn each of these windows into one Stream converted, where with mapToInt(Integer::intValue).average() the average value is calculated. This method provides one OptionalDouble, from which means orElse(0) a double is extracted if the window is empty (which is impossible in this use case because the window size is smaller or equal to the initial list).

The resulting List contains the average values ​​of the overlapping triples and looks specifically as follows:

[2.0, 3.0, 4.0]

The code’s declarative style promotes readability and understanding, while the Gatherers-API provides an elegant way to perform sliding aggregations directly in the stream context without resorting to manual window logic or imperative loops.

possible optimizations

Is Several optimization goals can be identified that can improve both the robustness against runtime errors and the performance and semantic readability of the code. Each of these goals brings different requirements for the design of the code.

Regarding performanceoptimization , it can be determined that the current source code creates unnecessary objects by repeatedly creating temporary ones, such as stream objects within the mapping. This can lead to a noticeable increase in the garbage collection load, especially with large amounts of data. To increase performance, you should ensure repeated use of Stream () within the window and instead directly with the existing one List work provided through windowSliding. An alternative solution could look like this:

xmlCopy

List<Double> result = input.stream() .gather(Gatherers.windowSliding(3)) .map(window -> { int sum = 0; for (int i : window) sum += i; return sum / 3.0; }) .toList();

This version completely eliminates the overhead of internal stream processing per window. Instead, the summation is done directly via a simple loop. This not only reduces the allocation of temporary objects, but allows the JVM to perform aggressive inlining and loop unrolling optimizations. In addition, the division by the window size is constant, since windowSliding(3) a fixed window size guaranteed. However, if the window size varies dynamically, you could alternatively window.size() determine at runtime without endangering semantic correctness.

Regarding thereadability , It should be noted that functional elegance and imperatively expressed clarity are not necessarily in contradiction. The original version with mapToInt(…).average().orElse(0) Although it is syntactically compact, it is less immediately understandable for many developers, especially with regard to the treatment of OptionalDouble. The imperative variant with direct summation and division, on the other hand, clearly reveals the underlying intention - the calculation of an average over three elements - without a deep understanding of the Stream-API is required. For teams with heterogeneous levels of experience or in code bases with a strong focus on maintainability, the following version is often preferable:

xmlCopy

List<Double> result = input.stream() .gather(Gatherers.windowSliding(3)) .map(window -> { int sum = 0; for (Integer value : window) { sum += value; } return (double) sum / window.size(); }) .toList();

This variant avoids potentially confusing method chains and expresses the calculation step by step. This not only makes the code easier to understand, but also easier to test, as each calculation step can be validated individually. At the same time, it remains compact and uses modern Java constructs such as Stream.gather() in an idiomatic way.

Overall, it can be seen that by taking robustness, performance and readability into account, both the functional and structural quality of the code can be increased. The choice of specific optimization always depends on the application context and the requirements for runtime behavior, error handling and maintainability.

That should be it at this point for now. In the following post I will go into more detail about the practical use of streams, the use of gatherers and similar things.

Happy Coding

Sven

]]>JavaStreamshttps://svenruppert.com/posts/tornadovm-boosting-the-concurrency/Sat, 23 Nov 2024 19:40:08 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/tornadovm-boosting-the-concurrency/TornadoVM is an open-source framework that extends the Java Virtual Machine (JVM) to support hardware accelerators such as Graphics Processing Units (GPUs), Field-Programmable Gate Arrays (FPGAs), and multi-core central processing units (CPUs). This allows developers to accelerate their Java programs on heterogeneous hardware without needing to rewrite their code in low-level languages such as CUDA or OpenCL.<![CDATA[

TornadoVM is an open-source framework that extends the Java Virtual Machine (JVM) to support hardware accelerators such as Graphics Processing Units (GPUs), Field-Programmable Gate Arrays (FPGAs), and multi-core central processing units (CPUs). This allows developers to accelerate their Java programs on heterogeneous hardware without needing to rewrite their code in low-level languages such as CUDA or OpenCL.

1. Motivation and Background
2. How TornadoVM Works
3. Key Features
4. Use Cases
5. Performance and Benchmarks
6. Limitations and Challenges
7. Integration with Java Ecosystem
8. Future Directions
9. Theoretical Background: Parallelism in Matrix Multiplication
10. How TornadoVM Exploits Parallelism
11. Example: Matrix Multiplication in Java with TornadoVM 1.Code Explanation
12. Theoretical Discussion
13. Performance Considerations

Here’s an overview of TornadoVM’s key aspects and features:

Motivation and Background

The growth in heterogeneous computing, driven by the increasing demand for performance in fields like machine learning, data processing, and scientific computing, has led to the development of specialised hardware such as GPUs and FPGAs. These devices offer significant performance improvements over traditional CPUs for parallel workloads.

However, programming for these accelerators traditionally requires low-level programming models like CUDA or OpenCL, which are less developer-friendly than higher-level languages like Java. TornadoVM addresses this challenge by allowing developers to use the Java programming language while transparently offloading computations to hardware accelerators.

The JVM has been traditionally optimised for multi-core CPUs, but its design is unsuited for exploiting massively parallel architectures like GPUs. TornadoVM fills this gap by introducing a runtime and compilation infrastructure that allows the JVM to leverage these devices.

How TornadoVM Works

TornadoVM provides an intermediate layer between Java applications and the underlying hardware. It intercepts certain portions of the Java code, such as loops and parallelizable methods, and compiles them to run on GPUs, FPGAs, or multi-core CPUs.

The TornadoVM runtime is divided into three main components:

Bytecode Analyzer : TornadoVM analyses the Java bytecode to identify portions of the code that can be accelerated. This typically includes loops, data-parallel operations, and compute-intensive methods.

JIT Compiler : TornadoVM uses a Just-In-Time (JIT) compiler to translate the selected portions of the bytecode into OpenCL or PTX (NVIDIA’s parallel thread execution format) code that can run on GPUs or FPGAs. The compiler also optimises the code for the specific hardware target.

Task Scheduling and Execution : Once compiled, TornadoVM schedules the execution tasks on the available hardware. It manages data transfers between the host (CPU) and the accelerators and ensures the results are correctly integrated into the running Java program.

Key Features

Java API : TornadoVM provides a simple Java API that allows developers to annotate the code they want to accelerate. This can include compute-intensive methods or loops suitable for parallel execution.

Automatic Offloading : TornadoVM automatically detects which portions of the code can be offloaded to the hardware accelerators. The developer doesn’t need to manage low-level details such as memory transfers or kernel execution.

Multi-backend Support : TornadoVM supports multiple backends, including OpenCL, PTX (for NVIDIA GPUs), and SPIR-V. This enables it to target various hardware platforms, from different GPU vendors (NVIDIA, AMD, Intel) to FPGAs.

Data Management : TornadoVM handles data transfers between the main memory (used by the CPU) and the memory of the accelerators (such as the GPU memory). It uses a memory management system to minimise unnecessary data transfers, reduce overhead, and improve performance.

Heterogeneous Task Scheduling : TornadoVM can execute multiple tasks across different devices in parallel. For example, it can execute one part of the code on a CPU and another part on a GPU, enabling efficient use of all available hardware resources.

Use Cases

TornadoVM is particularly suited for applications that can benefit from parallel execution. Some of the most common use cases include:

Machine Learning : Many machine learning algorithms, especially those based on matrix operations (like neural networks), can be accelerated on GPUs or FPGAs using TornadoVM.

Big Data Processing : Frameworks like Apache Spark can be integrated with TornadoVM to accelerate data processing pipelines. By offloading certain operations to GPUs, it can significantly reduce the time required to process large datasets.

Scientific Computing : Many scientific applications involve complex mathematical computations that can benefit from parallel execution on hardware accelerators. TornadoVM makes it possible to run these applications on GPUs without rewriting the code in CUDA or OpenCL.

Financial Modeling : Algorithms used in financial simulations, such as Monte Carlo simulations or options pricing, often involve parallelizable computations that can be offloaded to hardware accelerators using TornadoVM.

Performance and Benchmarks

TornadoVM has been shown to improve performance significantly for various applications. For example, benchmarks have demonstrated that TornadoVM can speed up certain machine learning algorithms by 10x to 100x when running on GPUs, compared to the same code running on a CPU.

The performance gains are particularly significant for compute-bound applications that involve a large number of parallel operations. However, not all applications will benefit from TornadoVM. Applications that are I/O-bound or involve a lot of sequential processing may not see significant speedups.

Limitations and Challenges

Despite its benefits, TornadoVM has some limitations:

Device Availability : TornadoVM requires the presence of a hardware accelerator (such as a GPU or FPGA) to provide performance improvements. If no such device is available, the code will fall back to running on the CPU.

Limited Scope : TornadoVM is most effective for applications that involve parallelizable computations. Applications with complex control flow or those that are heavily reliant on I/O may not see significant performance improvements.

Hardware-Specific Tuning : While TornadoVM abstracts many of the details of hardware acceleration, developers may still need to fine-tune their code for specific hardware platforms to achieve optimal performance.

Integration with Java Ecosystem

One of the key strengths of TornadoVM is its seamless integration with the existing Java ecosystem. It works with the standard Java Development Kit (JDK) and can be integrated with popular Java frameworks such as:

Apache Spark : TornadoVM can be used to accelerate Spark workloads by offloading certain operations to hardware accelerators. This can significantly reduce the time required to process large datasets.

JVM-based Languages : TornadoVM can be used with any JVM-based language, including Scala, Kotlin, and Groovy. This makes it a versatile solution for developers working in various JVM languages.

GraalVM : TornadoVM can also be used in conjunction with GraalVM, an advanced JVM that includes an optimising compiler and support for polyglot programming. GraalVM can further enhance TornadoVM’s performance by applying additional optimizations at runtime.

Future Directions

TornadoVM is an actively developed project, and its developers are working on several features to improve its functionality and performance. Some of the key areas of future development include:

Expanded Hardware Support : TornadoVM continually supports new hardware platforms, including newer generations of GPUs and FPGAs.

Improved Optimization : The TornadoVM team is working on improving the JIT compiler to generate more optimised code for hardware accelerators, leading to even more significant performance gains.

Automatic Parallelization : While TornadoVM already supports automatic offloading of specific code segments, there is ongoing work to automate further the process of identifying parallelizable code and offloading it to hardware accelerators.

Deep Integration with Machine Learning Frameworks : TornadoVM is exploring deeper integration with machine learning frameworks like TensorFlow and PyTorch to provide hardware acceleration for a wider range of machine learning tasks.

TornadoVM is a powerful tool that brings hardware acceleration to the Java ecosystem, allowing developers to harness the power of GPUs, FPGAs, and multi-core CPUs without having to leave the familiar world of the JVM. By simplifying the process of offloading computations to heterogeneous hardware, TornadoVM opens up new possibilities for accelerating a wide range of applications, from machine learning and big data processing to scientific computing and financial modelling. As the demand for high-performance computing continues to grow, TornadoVM is likely to play an increasingly important role in enabling Java developers to take full advantage of modern hardware platforms.

Let’s have an example

To give you a comprehensive example in Java using TornadoVM, we will look at simple matrix multiplication, a common parallelisable operation. The example will include a theoretical background on parallelism, how TornadoVM exploits it, and how the program is structured to benefit from GPU/FPGA acceleration.

Theoretical Background: Parallelism in Matrix Multiplication

Matrix multiplication is a good example of parallel execution because each element of the result matrix can be computed independently of the others. Consider two matrices,A andB, where:

- MatrixA is of size N x M,

- MatrixB is of size M x K.

The resulting matrixC will be of size N x K, where each element C[i][j] is the dot product of the i-th row ofA and the j-th column ofB:

This operation involves a large number of independent computations, meaning that all elements ofC can be computed in parallel. This is an ideal scenario for hardware acceleration using GPUs or FPGAs, which can perform a massive number of operations simultaneously.

How TornadoVM Exploits Parallelism

TornadoVM can accelerate the matrix multiplication by identifying the parallelisable loops (i.e., loops where the computations are independent). It translates the loops into tasks that can be distributed across GPU cores. The key steps are:

Bytecode Analysis : TornadoVM inspects the Java bytecode to identify parallelisable sections, such as loops where each iteration performs independent operations.

JIT Compilation : The identified sections are compiled Just-In-Time (JIT) into code that runs on the accelerator (such as PTX for NVIDIA GPUs).

Task Execution : The compiled tasks are executed on the available hardware, while data is transferred between the host (CPU) and the accelerator as needed.

Example: Matrix Multiplication in Java with TornadoVM

Below is a Java code example that uses TornadoVM to perform matrix multiplication.

javaCopy

importuk.ac.manchester.tornado.api.TaskSchedule;importuk.ac.manchester.tornado.api.TornadoVM;importuk.ac.manchester.tornado.api.annotations.Parallel;publicclassMatrixMultiplication{    // Matrix multiplication using TornadoVMpublicstaticvoidmultiplyMatrix(float[][]A,float[][]B,float[][]C,intN,intM,intK){        // TornadoVM annotation to indicate parallelism        @Parallel        for(inti=0;i<N;i++){            for(intj=0;j<K;j++){                floatsum=0.0f;                for(intk=0;k<M;k++){                    sum+=A[i][k]*B[k][j];                }                C[i][j]=sum;            }        }    }    publicstaticvoidmain(String[]args){        intN=1024;// Number of rows in A        intM=1024;// Number of columns in A (and rows in B)        intK=1024;// Number of columns in B        // Initialize matrices        float[][]A=newfloat[N][M];        float[][]B=newfloat[M][K];        float[][]C=newfloat[N][K];// Result matrix        // Fill matrices A and B with random values        for(inti=0;i<N;i++){            for(intj=0;j<M;j++){                A[i][j]=(float)Math.random();            }        }        for(inti=0;i<M;i++){            for(intj=0;j<K;j++){                B[i][j]=(float)Math.random();            }        }        // Create a task schedule for TornadoVM        TaskSchedulets=newTaskSchedule("s0")            .task("t0",MatrixMultiplication::multiplyMatrix,A,B,C,N,M,K)            .mapAllTo(TornadoVM.getDefaultDevice());        // Execute the task on the GPU        ts.execute();        // Print out a few results for verification        System.out.println("Result (C[0][0]): "+C[0][0]);        System.out.println("Result (C[10][10]): "+C[10][10]);    }}

Code Explanation

Matrix Initialization :

MatricesA andB are initialised with random floating-point values. The matrixC will store the result of the multiplication.

Matrix Multiplication Function :

ThemultiplyMatrix method multiplies matrixA by matrixB and stores the result in matrixC. The@Parallel annotation tells TornadoVM that this loop is parallelisable, meaning that each computation ofC[i][j] can be executed independently across different threads or hardware cores.

Task Scheduling with TornadoVM :

TheTaskSchedule object schedules a TornadoVM task, where themultiplyMatrix function is executed..mapAllTo(TornadoVM.getDefaultDevice()) specifies that the entire task will be mapped to the default TornadoVM device, which could be a GPU, FPGA, or multi-core CPU, depending on the available hardware. The.execute() method runs the task, performing the matrix multiplication on the selected hardware accelerator.

Performance Consideration :

For large matrix sizes (e.g., 1024x1024, as used in the example), the GPU can perform the multiplication much faster than a CPU by exploiting parallelism. TornadoVM handles the data transfer and task offloading, ensuring efficient execution on the accelerator.

Theoretical Discussion

Parallelism :

Each element C[i][j] is independent of all other elements in matrix multiplication, making the operation embarrassingly parallel. The GPU can execute these independent operations concurrently with its thousands of cores, leading to significant speedups over a sequential CPU-based implementation.

Data Transfer :

TornadoVM manages data transfers between the CPU (host) and the GPU (device). Before the computation starts, matrices A and B are copied to the GPU’s memory. After the computation, the resulting matrixC is transferred back to the CPU. TornadoVM optimised these transfers, ensuring that only the necessary data is copied and avoiding redundant transfers to minimise overhead.

Memory Coalescing :

Memory access patterns are important for performance on a GPU. TornadoVM attempts to optimise memory access through techniques like memory coalescing, where adjacent threads access adjacent memory locations, reducing the number of memory transactions and improving bandwidth utilisation.

Loop Parallelization :

The loop structure is a perfect candidate for parallelisation: TornadoVM splits the outer two loops (overi' andjindices) across different GPU threads. Each thread computes a single element of the result matrixC, while the innermost loop (overk`) can be executed within each thread.

Just-in-Time Compilation :

TornadoVM’s JIT compiler generates GPU-specific code (such as PTX code for NVIDIA GPUs) based on the annotated Java bytecode. This process occurs at runtime, allowing TornadoVM to optimise the code for the specific hardware platform in use.

Performance Considerations

Using TornadoVM on a GPU or FPGA for matrix multiplication can yield significant performance improvements, especially for large matrices. For instance:

A CPU may take several seconds to multiply two large matrices (e.g., 1024x1024), whereas a GPU can perform the same operation in a fraction of the time by leveraging its thousands of cores. The larger the matrices, the more efficient TornadoVM becomes, as the overhead of transferring data between the CPU and the accelerator becomes less significant relative to the computation time.

This example illustrates how TornadoVM allows Java developers to exploit hardware accelerators like GPUs for computationally expensive tasks like matrix multiplication. By identifying parallelisable loops and offloading them to the accelerator, TornadoVM achieves substantial performance improvements while allowing developers to continue working in Java without needing lower-level languages like CUDA or OpenCL.

Happy Coding

Sven

]]>JavaUncategorizedhttps://svenruppert.com/posts/cache-poisoning-attacks-on-dependency-management-systems-like-maven/Wed, 13 Nov 2024 14:15:16 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cache-poisoning-attacks-on-dependency-management-systems-like-maven/Cache poisoning on Maven Caches is a specific attack that targets how Maven Caches manages packages and dependencies in a software development process. It’s essential to understand how Maven works before we look at the details of cache poisoning.<![CDATA[

Cache poisoning on Maven Caches is a specific attack that targets how Maven Caches manages packages and dependencies in a software development process. It’s essential to understand how Maven works before we look at the details of cache poisoning.

1. Overview of Maven and its caches
2. What is cache poisoning?
3. Types of cache poisoning on Maven caches
   1. Man-in-the-Middle (MITM) Cache Poisoning
   2. Exploit repository vulnerabilities
   3. Dependency Confusion
      1. Basics of Dependency Confusion
      2. How Dependency Confusion Was Discovered
      3. Why does Dependency Confusion work so effectively?
   4. Typosquatting
      1. Basics of typosquatting
      2. Typical typosquatting techniques
      3. Dangers of typosquatting
      4. Maven typosquatting cases
4. Process of a cache poisoning attack on Maven
5. Security mechanisms to protect against cache poisoning
   1. Regular updates and patch management
   2. Using HTTPS
   3. Signature verification
   4. Improve repository security
   5. Dependency Scanning and Monitoring
   6. Version Pinning
   7. Private Maven-Repositories
   8. Implementation of code reviews and security checks
6. Understanding CVEs in the context of Maven and cache poisoning
7. Relevant CVEs related to Maven and repository management tools
   1. CVE-2020-13949: Remote Code Execution in Apache Maven
   2. CVE-2021-25329: Denial of Service in Maven Artifact Resolver
   3. CVE-2019-0221: Directory Traversal in Sonatype Nexus Repository Manager
   4. CVE-2022-26134: Arbitrary File Upload in JFrog Artifactory
   5. CVE-2021-44228: Log4Shell (Log4j)
8. Analysis and impact of the mentioned CVEs
9. Future challenges and continuous security improvements
10. Conclusion

Overview of Maven and its caches

Apache Maven is a widely used build management tool in Java projects. It automates the dependency management, build process, and application deployment. However, some fundamental mechanisms make it necessary to consider the security of repositories and dependencies when using Maven.

Maven uses repositories to manage libraries and dependencies. There are two types of Maven repositories:

Local repository : A copy of all downloaded libraries and dependencies is saved on the local machine.

Remote Repositories : Maven can access various remote repositories, such as the central Maven repository or a company’s custom repositories.

After downloading them from a remote repository, Maven stores all dependencies in the local repository (cache). This allows dependencies that are needed multiple times to load more quickly because they do not have to be downloaded from a remote repository each time.

What is cache poisoning?

Cache poisoning is a class of attacks in which an attacker fills a system’s cache (in this case Maven caches) with manipulated or malicious content. This results in legitimate requests that should receive the original data instead of the data injected by an attacker. In Maven terms, cache poisoning refers to when an attacker injects malicious artefacts into a developer’s or build’s cache by exploiting a vulnerability in the Maven build process or repository servers.

The attack aims to deliver malicious dependencies that are then integrated into software projects. These poisoned dependencies could contain malicious code to steal sensitive data, take control of the system, or sabotage the project.

Types of cache poisoning on Maven caches

There are several scenarios in which cache poisoning attacks can be carried out on Maven repositories:

Man-in-the-Middle (MITM) Cache Poisoning

A man-in-the-middle attack allows an attacker to intercept and manipulate network traffic between the developer and the remote Maven repository. If communication is not encrypted, an attacker can inject crafted artefacts and introduce them into the local Maven cache. As a result, the developer believes that the dependencies come from a trusted repository, when in fact, they have been tampered with.

Such an attack could be successful if Maven communicates with repositories over unsecured HTTP connections. The central Maven repository (Maven Central) now exclusively uses HTTPS to prevent such attacks, but some private or legacy repositories use HTTP.

Exploit repository vulnerabilities

If an attacker gains access to the remote repository, they can upload arbitrary artefacts or replace existing versions. This happens, for example, if the repository is poorly secured or a vulnerability in the repository management tool (like Nexus or Artifactory) is exploited. In this case, the attacker can inject malware directly into the repository, causing developers worldwide to download the compromised artefact and store it in their Maven cache.

Dependency Confusion

A particularly dangerous attack vector that has received much attention in recent years is the so-called “dependency confusion” attack. This attack is because many modern software projects draw dependencies from internal and private repositories and public repositories such as Maven Central. The main goal of a Dependency Confusion attack is to inject malicious packages via publicly accessible repositories into a company or project that believes it is using internal or private dependencies.

Basics of Dependency Confusion

Many companies and projects maintain internal Maven repositories where they store their own libraries and dependencies that are not publicly accessible. These internal libraries can implement specific functionalities or make adaptations to public libraries. Developers often define the name and version of dependencies in the Maven configuration (pom.xml) without realising that Maven prioritises dependencies, favouring public repositories like Maven Central over internal ones unless explicitly configured otherwise.

A dependency confusion attack exploits exactly this priority order. The attacker publishes a package with the same name as an internal library to the public Maven repository, often with a higher version number than the one used internally. When Maven then looks for that dependency, it usually prefers the publicly available package rather than the private internal version. This downloads the malicious package and stores it in the developer’s Maven cache, from where it will be used in future builds.

How Dependency Confusion Was Discovered

A security researcher named Alex Birsan popularised this attack in 2021 when he demonstrated how easy it was to poison dependencies in projects at major tech companies. By releasing packages with the same names as internal libraries of large companies such as Apple, Microsoft, and Tesla, he successfully launched dependency confusion attacks against these companies.

Birsan did not use malicious content in his attacks but harmless code to prove that the system was vulnerable. He was able to show that in many cases, the companies’ build systems had downloaded and used the malicious (in his case harmless) package instead of the real internal library. This disclosure led to massive awareness in the security community about the risks of Dependency Confusion.

Why does Dependency Confusion work so effectively?

The success of a Dependency Confusion attack lies in the default configuration of many build systems and the way Maven resolves dependencies. There are several reasons why this attack vector is so effective:

• - Automatic prioritisation of public repositories
• - Trust the version number
• - Missing signature verification
• - Reliance on external code

Typosquatting

Typosquatting is an attack technique that exploits user oversight by targeting common typos that can occur while typing package names in software development, such as in Maven. Attackers release packages with similar or slightly misspelt names that closely resemble legitimate libraries. When developers accidentally enter the wrong package name in their dependency definitions or automated tools resolve these packages, they download the malicious package. Typosquatting is one of the most well-known attack methods for manipulating package managers, such as Maven, npm, PyPI, and others that host publicly available libraries.

Basics of typosquatting

Typosquatting is based on the idea that users often make typos when entering commands or package names. Attackers exploit this by creating packages or artifacts with names that are very similar to well-known and widely used libraries but differ in small details. These details may include minor variations such as missing letters, additional characters, or alternative spellings.

Typical typosquatting techniques

Misspelled package names :

One of the most straightforward techniques is to change or add a letter in the name of a well-known library. An example would be the package**com.google.common**, which is often used. An attacker could use a package named**com.gooogle.common** (with an extra “o”) that is easily overlooked.

Different spellings :

Attackers can also use alternative spellings of well-known libraries or names. For example, an attacker could use a package named**com.apache.loggin** to publish the popular**com.apache.logging** looks similar, but due to the missing letter combination “g " and “n " at “logging " is easily overlooked.

Use of prefixes or suffixes :

Another option is to add prefixes or suffixes that increase the similarity to legitimate packages. For example, an attacker could use the package**com.google.common-utils** or**com.google. commonx to publish** the same as the legitimate package**com.google.common** resembles.

Similarity in naming :

Attackers can also take advantage of naming conventions in the open-source community by publishing packages containing common terms or abbreviations often used in combination with other libraries. An example would be releasing a package like**common-lang3-utils**, which is linked to the popular Apache Commons library `commons-lang3 ’ remembers.

Dangers of typosquatting

The threat of typosquatting is grave because it is difficult to detect. Developers often rely on their build tools like Maven to reliably download and integrate packages into their projects. If an incorrect package name is entered, you may not immediately realise that you have included a malicious dependency. Typosquatting is a form of social engineering because it exploits people’s susceptibility to errors.

A successful typosquatting attack can lead to severe consequences:

• Data loss
• Malware injection
• Loss of trust

Maven typosquatting cases

There have also been incidents of typosquatting in the Maven community. In one case, a package named**commons-loggin** was published, corresponding to the legitimate Apache Commons logging package**commons-logging**. Developers who entered the package name incorrectly downloaded and integrated the malicious package into their projects, creating potential security risks.

Typosquatting is a sophisticated and difficult-to-detect attack method that targets human error. Attackers take advantage of the widespread use of package managers such as Maven, npm, and PyPI by publishing slightly misspelt or similar-sounding packages that contain malicious code. Developers and organisations must be aware of this threat and take appropriate protective measures to ensure that only legitimate and trustworthy packages are included in their projects.

Process of a cache poisoning attack on Maven

A typical sequence of a cache poisoning attack on Maven could look like this:

Identification of a target repository : The attacker is looking for a Maven repository used by developers but may have vulnerabilities. This can happen, for example, through outdated versions of publicly available repository management tools.

Handling Artifacts : The attacker manipulates the artefact, e.g. a JAR file, by adding malicious code. This can range from simple backdoors to complex Trojans.

Provision of the poisoned artefact : The manipulated artefact is either uploaded to the public repository (e.g., in the form of a typosquatting package) or injected directly into a compromised target repository.

Download by developer : The developer uses Maven to update or reload the dependencies for his project. Maven downloads the poisoned artefact, which is stored in the local cache.

Compromising the project : Maven will use the poisoned artefact from the cache in future builds. This artefact can then execute malicious code in the application’s context, resulting in system compromise.

Security mechanisms to protect against cache poisoning

Various measures should be implemented on both the developer and repository provider sides to protect against cache poisoning attacks.

Regular updates and patch management

Make sure Maven, its plugins, and all repository management tools are always up to date. Security updates should be applied immediately to address known vulnerabilities.

Using HTTPS

The use of encrypted connections (HTTPS) between Maven and the repository is crucial to ensure that no man-in-the-middle attacks can be performed on the transferring artifacts. Maven Central enforces HTTPS connections, but private repositories should also adhere to this standard.

Signature verification

Another protective measure is the use of cryptographic signatures for artefacts. Maven supports the use of PGP signatures to ensure the integrity of artefacts. Developers should ensure that the signatures of downloaded artefacts are verified to ensure that they have not been tampered with.

Improve repository security

Repository providers should ensure that their repositories are well protected by implementing robust authentication mechanisms, regular patches and updates to repository management tools such as Nexus or Artifactory.

Dependency Scanning and Monitoring

Tools like OWASP Dependency Check or Snyk can scan known dependency vulnerabilities. These tools can help identify malicious or stale dependencies and prevent them from entering the Maven cache.

Version Pinning

“Version pinning” means setting specific versions of dependencies in thepom.xml file instead of using dynamic version ranges ([1.0,)). This helps prevent unexpected updates and ensures that only explicitly defined versions of the artefacts are used.

Private Maven-Repositories

One approach to maintaining control over dependencies is maintaining a private Maven repository within the organisation. This ensures that only checked artifacts end up in the internal cache, reducing the risk of introducing malicious dependencies into the build process.

Implementation of code reviews and security checks

Conduct regular code reviews to ensure only trusted dependencies are used in projects. Automated security checks can provide additional security.

Understanding CVEs in the context of Maven and cache poisoning

CVE (Common Vulnerabilities and Exposures) is a standardised system for identifying and cataloguing security vulnerabilities in software. Each CVE number refers to a specific vulnerability discovered and documented by security experts.

There are no specific CVEs that exclusively target cache poisoning in the context of Maven and cache poisoning. Instead, various vulnerabilities in Maven itself, its plugins, or the underlying repository management tools (such as Sonatype Nexus or JFrog Artifactory) can be indirectly exploited for cache poisoning attacks. These vulnerabilities could allow attackers to manipulate dependencies, compromise the integrity of downloads, or bypass Maven’s security mechanisms.

Relevant CVEs related to Maven and repository management tools

Although no CVEs are explicitly classified as cache poisoning, there are several vulnerabilities in Maven and related tools that could potentially be exploited for cache poisoning attacks:

CVE-2020-13949: Remote Code Execution in Apache Maven

• Description : This vulnerability affected Maven Surefire plugin versions prior to 2.22.2, which could enable remote code execution (RCE). An attacker could use specially crafted POM files to execute malicious code on the build system.
• Relevance to cache poisoning : By running RCE, an attacker could inject crafted dependencies into the local Maven cache, which could compromise future builds.
• Reference : CVE-2020-13949

CVE-2021-25329: Denial of Service in Maven Artifact Resolver

• Description : This vulnerability affected the Apache Maven Artifact Resolver component, which is responsible for resolving and downloading artefacts. A specially crafted POM file could lead to a denial of service (DoS).
• Relevance to cache poisoning : A DoS attack could impact the availability of Maven repositories, forcing developers to use alternative (possibly insecure) repositories, increasing the risk of cache poisoning.
• Reference : CVE-2021-25329

CVE-2019-0221: Directory Traversal in Sonatype Nexus Repository Manager

• Description : This vulnerability allows attackers to access files within the Nexus Repository Manager through a directory traversal attack.
• Relevance to cache poisoning : By accessing critical files or configurations, attackers could compromise the repository manager and insert malicious artefacts, which are then downloaded by developers and stored in the local Maven cache.
• Reference : CVE-2019-0221

CVE-2022-26134: Arbitrary File Upload in JFrog Artifactory

• Description : This vulnerability allowed attackers to upload arbitrary files to a JFrog Artifactory server, which could result in a complete compromise of the server.
• Relevance to cache poisoning : By uploading arbitrary files, attackers could inject malicious Maven artefacts into the repository, which developers then download and cache.
• Reference : CVE-2022-26134

CVE-2021-44228: Log4Shell (Log4j)

• Description : This widespread vulnerability affected the Log4j library and allowed remote code execution by exploiting JNDI injections.
• Relevance to cache poisoning : Many Maven projects use Log4j as a dependency. A manipulated version of this dependency could enable RCE through cache poisoning.
• Reference : CVE-2021-44228

Analysis and impact of the mentioned CVEs

The above CVEs illustrate how vulnerabilities in Maven and related tools can potentially be exploited for cache poisoning and other types of attacks:

• Remote Code Execution (RCE) : Vulnerabilities such as CVE-2020-13949 and CVE-2021-44228 allow attackers to execute malicious code on the build system. This can be used to inject manipulated dependencies into the local cache.
• Denial of Service (DoS) : CVE-2021-25329 demonstrates how attackers can impact the availability of Maven repositories. This may result in developers being forced to resort to alternative sources that may be unsafe.
• Directory Traversal and Arbitrary File Upload : CVE-2019-0221 and CVE-2022-26134 demonstrate how attackers can compromise repository management tools to upload malicious artefacts that developers then unknowingly use in their projects.

Future challenges and continuous security improvements

As the use of open-source libraries in modern software development continues to increase, the threat of attacks such as cache poisoning also increases. Automating the build process and relying on external repositories creates an expanded attack surface that attackers seek to exploit.

It is becoming increasingly important for companies and developers to cultivate a security culture prioritising the safe handling of dependencies and artefacts. This requires not only the implementation of technical protection measures but also the training of developers to raise awareness of the risks of cache poisoning and other dependency attacks.

Conclusion

Cache poisoning attacks on Maven caches are a severe risk, especially at a time when open-source components play an essential role in software development. The attacks exploit vulnerabilities in how dependencies are resolved, downloaded and cached.

Developers and companies must implement best security practices to prevent such attacks. This includes using HTTPS, signing and verifying artefacts, securing repositories, regular vulnerability scanning, and controlling the dependencies introduced into their projects.

Awareness of the risks and implementing appropriate security measures are key to preventing cache poisoning attacks and ensuring the integrity of software development processes.

Happy Coding

Sven

]]>JavaSecure Coding PracticesSecurityToolsUncategorizedhttps://svenruppert.com/posts/junit-annotations-in-focus-the-connection-between-test-and-testable/Fri, 08 Nov 2024 09:12:39 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/junit-annotations-in-focus-the-connection-between-test-and-testable/The annotations @Test and @Testable have played an important role in the Java ecosystem regarding application testing. They are essential tools that help developers make unit and automated testing more effective. In this paper, we will explore the differences and connections between @Test and @Testable analyze and the motivation behind the introduction of @Testable understand. We will also play the role of @Testable in developing your test engines and discuss their importance for the flexibility and expandability of tests.<![CDATA[

The annotations @Test and @Testable have played an important role in the Java ecosystem regarding application testing. They are essential tools that help developers make unit and automated testing more effective. In this paper, we will explore the differences and connections between @Test and @Testable analyze and the motivation behind the introduction of @Testable understand. We will also play the role of @Testable in developing your test engines and discuss their importance for the flexibility and expandability of tests.

1. @Test and their usage
2. @Testable - What is it for?
3. The connection between @Test and @Testable
4. The role of @Testable for developing your own TestEngines
5. Conclusion

@Test and their usage

Die Annotation @Test is familiar to most Java developers involved in unit testing. It comes from the JUnit framework, one of Java’s most commonly used frameworks for automated testing. @Test is used to identify methods as test methods that are then executed by the testing framework. When using JUnit 4 or 5, developers mark a method with @Testto, explicitly indicating that this method represents a test method to be tested. The JUnit framework takes over the execution of these methods and evaluates whether the expected conditions are met or errors occur. The main advantage of using @Test is that the framework automatically recognises the test methods and reports their results, including errors or successful runs. This makes maintaining and expanding tests easier, as all @Test-annotated methods can be executed without manual configuration.

An example of a simple test method looks like this:

javaCopy

importorg.junit.jupiter.api.Test;import staticorg.junit.jupiter.api.Assertions.assertEquals;publicclassCalculatorTest{    @Test    voidtestAddition(){        Calculatorcalc=newCalculator();        assertEquals(5,calc.add(2,3));    }}

In this example, the method testAddition is paired with @Test, so the JUnit framework recognises it as a test method. Executing the test checks whether the addition works correctly. The goal is to ensure that the tested code exhibits the desired behaviour, which increases the application’s reliability.

The use of @Test makes the testing process easy and efficient. Developers can create many test methods automatically recognised and executed by the JUnit framework. This is particularly useful in large projects where test coverage is essential in ensuring that the codebase remains stable and bug-free. @Test allows developers to quickly receive feedback on the health of their applications, resulting in shorter development cycles and improved software quality.

@Testable - What is it for?

@Testable comes from the packageorg.junit.platform.commons.annotation and was developed specifically for the JUnit 5 platform. The annotation @Testable is used to mark test methods that should be recognised by runtime environments such as test engines. The annotation @Test in JUnit 5 is a kind of special case @Testable-Annotation. In JUnit 5 is @Test yourself with @Testable annotated. This means that everyone with @Test marked method implicitly to @Testable is what ensures detection by testing engines. The use of @Testable in the annotation @Test ensures that all JUnit testing methods are automatically accessible to the testing platform without the need for additional annotations. This increases testability in different test environments and supports the modularity of JUnit 5.

The introduction of @Testable brought more flexibility to the test system. While @Test mainly specifies the concrete implementation of the test, @Testable is a common annotation that enables other testing engines to discover and execute test methods dynamically. This is particularly useful for facilitating integration between different testing tools and improving the extensibility of the JUnit 5 framework. Developers who need to perform different types of testing in a project benefit from the flexibility of @Testable because it enables different test engines to work together.

An example of using @Testable is rarely found at the developer level because the JUnit engine @Testable is used internally by all, with @Test automatically marking annotated methods as testable. This means developers usually don’t explicitly care about the annotation @Testable, as the runtime environment and the test engine handle this. This abstraction is advantageous because it saves developers the effort of explicitly ensuring their testing methods are discoverable - the platform does that for them.

The connection between @Test and @Testable

The main difference between @Test and @Testable lies in their application and purpose. While @Test is the specific testing method for the JUnit framework @Testable a general meta-annotation that helps test runtime environments better recognise and process test methods. @Testable provides an abstraction layer that ensures that any method marked as a test is also available to the testing engine. This mainly helps when developing frameworks and tools based on the JUnit platform.

JUnit 5 is a significant evolution compared to JUnit 4. It uses a more modular architecture and a testing engine called the “JUnit Platform”. This platform is designed to allow tests to be executed more flexibly so that other testing frameworks can also use the platform. @Testable plays an important role in this context as it ensures the visibility of the tests on the platform. Developers benefit from a simplified integration of test tools, as the platform contains all tests @Testable that are annotated, recognised and executed regardless of the test framework used.

The introduction of @Testable is, therefore, in the context of JUnit 5’s extensibility and flexibility. While JUnit 4 was primarily designed for unit testing, JUnit 5 enables better integration with other testing and tools. This makes the testing infrastructure for modern Java applications more powerful and adaptable. Using different testing engines and frameworks on the same platform significantly improves application testability and maintainability.

The role of @Testable for developing your own TestEngines

Die Annotation @Testable is crucial for developers who want to develop their testing engines. It provides a standardised way to mark test methods for the JUnit Platform, meaning that test engines compatible with it can recognise and execute test methods without making any special customisations. Using @Testable, Developers can ensure that their tests are recognised regardless of the testing framework used, promoting interoperability and extensibility of test engines.

The JUnit Platform is designed to support various testing engines, significantly increasing developers’ flexibility. @Testable provides a type of “contract” that allows the JUnit Platform to identify and execute tests, whether they were built using JUnit, TestNG, or a custom testing engine. Developers can write their own test engines seamlessly integrated into the JUnit Platform and benefit from the existing functionalities. This does @Testable, a central building block for the development of an expandable and modular test environment.

Another advantage of using @Testable for your own test engines is the standardisation of the test process. By labelling all test methods with @Testable, Developers can ensure that their tests run consistently across different projects and teams. This not only promotes the reusability of test engines but also the ability to develop new test engines that meet specific requirements without disrupting the fundamental architecture of the JUnit Platform.

With @Testable It also becomes possible to advance the development of specialized testing engines optimized for specific types of testing. For example, performance testing, security testing or end-to-end testing could be carried out by purpose-built engines based on the JUnit Platform. This creates a modular and versatile testing environment that allows the best tools to be used for different testing needs without compromising integration and compatibility.

Conclusion

The annotations @Test and @Testable play different but complementary roles in the Java testing ecosystem. @Test identifies specific testing methods for JUnit, while @Testable ensures that these methods are accessible to testing engines on a more general level. The introduction of @Testable as part of JUnit 5 offers greater flexibility and makes it easier to expand the testing system, which meets the modern requirements for testing in the Java world.

Overall, both annotations show the continuous development of testing tools in Java to meet the growing requirements for maintainability, extensibility and integration. The introduction of @Testable highlights the trend towards a more flexible and expandable testing infrastructure that makes it possible to combine different test types and engines in a common platform. This allows developers to make testing more efficient and test their applications comprehensively and reliably. By combining @Test and @Testable, A robust test basis is created that supports the development process and sustainably improves the quality of software products.

]]>JavaJUnit5TDDToolshttps://svenruppert.com/posts/the-risks-of-mocking-frameworks-how-too-much-mocking-leads-to-unrealistic-tests/Tue, 05 Nov 2024 17:49:33 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/the-risks-of-mocking-frameworks-how-too-much-mocking-leads-to-unrealistic-tests/Extensive use of mocking frameworks such as Mockito in software development can lead to unrealistic tests. This is because mocking frameworks simulate dependencies of classes or methods in order to test them in isolation. However, when too many mock objects are used, the test often loses touch with reality, which can affect the validity and reliability of the tests. It is important to use mocking carefully to find the right balance between isolated testing and realistic simulation.<![CDATA[

Extensive use of mocking frameworks such as Mockito in software development can lead to unrealistic tests. This is because mocking frameworks simulate dependencies of classes or methods in order to test them in isolation. However, when too many mock objects are used, the test often loses touch with reality, which can affect the validity and reliability of the tests. It is important to use mocking carefully to find the right balance between isolated testing and realistic simulation.

A mock is a simulated object that imitates the behaviour of a real component without executing any actual logic. This is particularly useful when a dependency is difficult to test, for example, because it involves an external database or web service that may have states that are difficult to reproduce or where testing would be very time-consuming and costly. Mocking creates a simplified and manageable version of this dependency, allowing targeted and isolated unit testing to be carried out. Mocking can be a precious technique in these cases, especially for writing independent tests that do not require external infrastructure. It reduces the complexity of tests and makes them faster and more deterministic.

However, it becomes problematic when the logic of the tested class is heavily influenced by these dependencies, especially when these dependencies have complex logic or dynamic behaviour. For example, a class that reads data from a database might implement logic that responds to the available data contents. This dynamic nature often cannot be realistically replicated with mocks, as mock objects are usually limited to predefined, simple return values. As a result, the test can lose touch with the objective complexity and only inadequately reflect the actual application logic. Realistic tests are created that only consider some possible application behaviours in interaction with other components, resulting in incomplete test coverage.

Extensive use of mocks often results in tests being tailored very specifically to the simulated scenarios. In such cases, the tests become helpful only for the implemented interfaces and the programmed code rather than for the actual use cases, which may contain a variety of unexpected events and uncontrolled factors. This can lead to a false sense of security that the application is functioning correctly, even though essential real-world conditions still need to be considered. For example, network failures, errors in data transmission or inconsistent data sets could not be simulated in the mocks because the tests focus only on positive confirmation of the function. This means that critical errors that could occur in a natural environment are missed, leading to severe problems in the production environment. To combat this, mocking should be used judiciously, and tests should also be performed that incorporate real-world dependencies to ensure that the code works correctly under real-world conditions such as network problems, database failures, or changing data.

A concrete example of this would be an application that retrieves data from a REST API and processes it. In a typical scenario, this REST API could return different data sets, ranging from simple JSON responses to complex, nested structures. When you replace the REST API with a mock object, you only simulate a specific reaction, which is highly simplified and mostly represents the ‘happy path’. In reality, however, the REST API could react in various ways: data could be missing, the data structure could change, new fields could be added, or network problems and unexpected error messages could occur. Additionally, errors such as receiving an empty response or submitting an HTTP error (e.g. 500 or 404) are often problematic to represent when only a mock is used. Tests based only on mocks cannot adequately simulate these realities because they need to consider the multitude of possible scenarios and failure cases that can arise in actual operation. This leads to a false sense of security about the robustness of the code. If the code is only tested in a strictly controlled environment, without considering the real variability of the API and its potential failures, serious errors may appear in the production environment that were never previously detected. Using accurate integration tests that respond to actual API responses and errors would be much more realistic in this case and would help ensure application robustness under real-world conditions.

Another problem arises when the extensive use of mocks oversimplifies the dependencies. For example, consider an e-commerce application that performs inventory checking. In a real-world implementation, this inventory check can involve a variety of complex operations, such as synchronising inventory across multiple geographically distributed warehouses, honouring reservations for ongoing orders, or updating inventory in real-time. These tasks require sophisticated logic, potentially involving numerous systems and technologies, to ensure inventories are always accurate. When you fully mock a warehouse management class, these complex dependencies are replaced with simple, simulated return values. However, this means that the test loses connection to the real functioning of the system. There is no guarantee that synchronisation logic will work smoothly under real-world conditions, such as network delays or inconsistencies between warehouses. The complexity of the real system is never taken into account in the tests, so possible sources of error remain undetected. In a production environment, unforeseen problems could arise that could have been avoided with a more realistic test environment. These complex scenarios show that real integration tests are necessary to ensure that the interaction between the components works correctly, even under load conditions or in the event of errors in the dependencies.

Another problem arises when test implementations are created too specifically for the mock scenarios. It often happens that developers unconsciously tailor the mock configuration exactly to the logic being tested without taking into account the realistic spread of the data or the potential errors. This results in the tests not reflecting unforeseen events that may occur in a real environment. An example of this is a database query where the test always assumes a successful query because the mock was configured that way. The realistic error case of a database failure, a zero response or a timeout is not practised. This means that the corresponding error handling is noticed in the production environment once it has already had an impact, which can lead to critical problems. Such gaps in error handling could be avoided if tests included a more realistic and varied simulation of possible scenarios. Therefore, tests must also cover hostile and unpredictable scenarios to ensure the application is robust enough to handle errors and exceptions.

Another problem when using mocking frameworks is that the functionality of the mocks only sometimes follows the actual implementation. Often, a simplified or even incorrect replica of the real components is created, which only reflects some aspects of real behaviour. For example, when replicating an external service using a mock, certain side effects, state changes, or dependencies in the real implementation cannot be adequately considered. This can cause the test to pass even though the actual implementation would fail due to its complexity.

Let’s take an example of a Java application that accesses an external cache service to cache data. Suppose we have a classProductService that reads product information from a database and stores it in a cache to improve performance. In a unit test, the cache service method could easily be mocked to ensure theput() call runs correctly. The corresponding Java code could look like this:

javaCopy

import staticorg.mockito.Mockito.*;publicclassProductServiceTest{    @Test    publicvoidtestFetchProduct(){        CacheServicecacheService=mock(CacheService.class);        DatabaseServicedatabaseService=mock(DatabaseService.class);        ProductServiceproductService=newProductService(databaseService,cacheService);        Productproduct=newProduct("123","TestProduct");        when(databaseService.getProduct("123")).thenReturn(product);        productService.fetchProduct("123");        verify(cacheService,times(1)).put("123",product);    }}

This test verifies that the product is cached correctly. However, it is not tested here whether the cache mechanism really works as desired, e.g., whether the cache is actually used for repeated requests to avoid unnecessary database queries. Such an error could go undetected in production if, in reality, the cache service is not functioning properly due to a synchronisation issue or misconfiguration. In addition, concurrent accesses to the cache could lead to synchronisation problems that need to be taken into account by simple mocking.

This test, therefore, provides no guarantee that the cache logic will work correctly under real-world conditions such as high load or concurrent accesses. The mocks only represent a simplified version of reality without taking into account all the possible scenarios that could occur in a real environment. Therefore, in addition to unit tests that use mocks, it is crucial also to perform integration tests that ensure that the cache service works correctly under realistic conditions. A classic example would be a component that internally uses caching mechanisms to minimise the number of requests to an external service. A simple mock could ignore the caching mechanism, so the test only checks whether a request is sent correctly to the external service, but not whether the caching logic works as intended. This allows cache management bugs or inefficient implementations to go undetected. Mocks also often do not realistically represent competing accesses to a resource, which could lead to synchronisation problems in the real implementation. Therefore, it is essential to ensure that mocks are as close as possible to the actual implementation to ensure realistic testing scenarios.

An alternative to mocking is integration or end-to-end testing, which tests the actual components and dependencies. Integration testing helps ensure that the application’s various modules interact with each other as expected, while end-to-end testing verifies the overall system under realistic conditions. This does not use mocks but rather real databases, REST APIs and other services, which better simulate the complexity of the real environment.

Another alternative is so-called ‘contract testing’. Contract testing checks whether communication between two services occurs according to a defined specification. This allows both sides of the communication to be developed and tested independently without simulating the entire service. This ensures a certain level of security that both components work correctly with each other without resorting to extensive mocking.

In addition, test containers or in-memory databases can be used as alternatives to mocks, especially for database testing. Using tools like Testcontainers, real database instances can be deployed in Docker containers for testing so that real database logic is tested. In-memory databases like H2 in Java are also useful for making testing more realistic because they simulate actual data logic but without the cost and complexity of a real database connection.

Another alternative method is the use of so-called ‘fake’ objects. Unlike mocks, which only simulate behaviour, fakes implement a reduced version of the real logic. These fakes are not as complex as the real components but still offer more realism than simple mocks. An example of this would be a ‘fake’ database object that mimics basic storage operations in a simple data structure rather than actually connecting to a database. Fakes are particularly useful when you need a realistic but not fully functional, environment to test certain aspects of the application. They are more flexible than mocks and provide the ability to run through different scenarios without simulating the full complexity of real implementations.

In summary, excessive use of mocking frameworks like Mockito often results in tests not being realistic. The isolation of components enabled by mocking is useful for performing isolated unit testing. However, if too many aspects of the application are simulated, the tests lose the ability to represent the interaction of real components and the actual challenges that might arise in operation. There is then a risk that the software in real operation will not show the behaviour that the tests suggest. It, therefore makes sense to find a balance between mocking and integration tests in order to represent realistic scenarios adequately. At the same time, other techniques, such as contract testing or the use of fakes, should also be considered to ensure that the tests replicate the real environment as closely as possible, thus ensuring a robust and reliable application.

Happy Coding

Sven

]]>JavaTDDhttps://svenruppert.com/posts/what-is-cwe-1007-insufficient-visual-discrimination-of-homoglyphs-for-you-as-a-user/Mon, 04 Nov 2024 12:34:27 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/what-is-cwe-1007-insufficient-visual-discrimination-of-homoglyphs-for-you-as-a-user/The world of cybersecurity is full of threats, many of which are surprisingly subtle and challenging to detect. One such threat is the problem of so-called homoglyphs. CWE-1007, also known as “Insufficient Visual Distinction of Homoglyphs Presented to User”, is a vulnerability often used by attackers to deceive and compromise your systems or data. In this blog article, you will get a deep insight into CWE-1007, understand its mechanisms, and how to protect yourself from such attacks. We will discuss examples, technical challenges, and best practices that can help you as a developer understand and mitigate this threat.<![CDATA[

The world of cybersecurity is full of threats, many of which are surprisingly subtle and challenging to detect. One such threat is the problem of so-called homoglyphs. CWE-1007, also known as “Insufficient Visual Distinction of Homoglyphs Presented to User”, is a vulnerability often used by attackers to deceive and compromise your systems or data. In this blog article, you will get a deep insight into CWE-1007, understand its mechanisms, and how to protect yourself from such attacks. We will discuss examples, technical challenges, and best practices that can help you as a developer understand and mitigate this threat.

1. What are Homoglyphs?
2. CWE-1007: A detailed investigation
3. Examples for CWE-1007
4. Why is CWE-1007 dangerous?
5. Possible attack scenarios
6. Defence strategies against CWE-1007
7. Technical challenges in homoglyph recognition
8. Best practices for developers
9. Case Study: Homoglyph Domain Attack
10. Homoglyph recognition tools
11. Which CWEs are often used in conjunction with CWE-1007? 1.CWE-601: Open Redirect: 2.CWE-643: Improper Neutralization of Data within XPath Expressions: 3.CWE-79: Cross-Site Scripting (XSS): 4.CWE-20: Improper Input Validation: 5.CWE-77: Command Injection:
12. Example application in Vaadin Flow
13. Conclusion
14. Next Steps

What are Homoglyphs?

Before we delve into CWE-1007, we must understand what homoglyphs are. Homoglyphs are characters that look visually similar but represent different Unicode codes. This can involve different letters, numbers or symbols. A well-known example is the Latin letter “O” and the number “0”, which can look almost identical to the human eye. There are many other examples, such as “l” (small L) and “I” (capital i), or Cyrillic letters that look like Latin letters.

The visual similarity of homoglyphs is often exploited to deceive you. Attackers use such characters to create phishing websites, generate fake URLs, or corrupt code to make you believe you are dealing with a trustworthy resource. This is particularly problematic because we humans use visual patterns to make quick decisions and can more easily fall victim to such deceptions.

CWE-1007: A detailed investigation

CWE-1007 refers to the insufficient visual discrimination of homoglyphs when presented as a user. If a system cannot distinguish between similar-looking characters or alert you to them, this can lead to significant security risks. You could accidentally click on a malicious link, visit a fake domain, or trust a fraudulent command.

This vulnerability is particularly common when displaying URLs, usernames, or commands. For example, you could click on a URL that appears to be correct but actually uses homoglyphs to represent a fake website. This allows attackers to steal passwords, credit card information, or other sensitive data. This problem is compounded by widespread internet use, as you constantly interact with potentially harmful content.

Examples for CWE-1007

A typical example of this vulnerability is the use of fake domains that use homoglyphs. Let’s say you receive an email with a link to “paypa1.com” (which uses the number “1” instead of the letter “l”). Without careful consideration, you might think this is a legitimate link to the PayPal website. The same principle can also apply to usernames on social networks or even essential commands in a console.

Another example is the use of homoglyphs in source code. Attackers could insert fake characters into the code that look like legal characters but result in different functionality. This can be particularly dangerous in open-source projects or teams where multiple developers work on the same code and could miss similar characters. This leads to security vulnerabilities that can be exploited for attacks and poses a high risk to the integrity of the code.

An everyday example is a fake username on social networks. An attacker could create a username like “facebook_support” using the Cyrillic “е”, which looks visually similar to the Latin “e”. You might think it’s an official support channel and click on malicious links or reveal sensitive data.

Why is CWE-1007 dangerous?

CWE-1007 is dangerous because humans rely heavily on visual cues to make decisions. The human eye is trained to recognise patterns and process information quickly, but often without closely examining each character. Attackers can specifically exploit this weakness to deceive you.

The danger of this vulnerability is not only that you could fall for phishing attacks. It can also lead to more severe security breaches, such as stealing credentials, tampering with software, or inserting malicious code into seemingly legitimate projects. In addition, this deception can also cause financial damage because you trust fake websites and enter your payment information, which can then be misused.

Another danger is that companies can lose their reputation due to these vulnerabilities. If you repeatedly fall for counterfeit versions of a well-known brand, your trust in that brand can be severely affected. Attackers use this method to undermine user trust and specifically to maximise damage to companies.

Possible attack scenarios

Phishing attacks using fake URLs : Attackers can create a URL almost identical to a known website. You click on the link without noticing that the domain has a slight change (e.g. a Cyrillic letter instead of a Latin letter). This allows attackers to access sensitive information such as passwords or credit card details.

Code injection through fake characters : In complex software projects, homoglyphs can cause the code to execute differently than it seems at first glance. You or other developers may miss these signs, resulting in hard-to-find vulnerabilities in your code. This could be used to insert malicious functions only noticeable in the production environment.

Social engineering in social networks : An attacker could create a username almost identical to a trusted contact’s to deceive you and steal information. For example, someone could use the name “LinkedIn Support” with a slightly modified letter to trick you into giving up your login information or clicking on malicious links. The deception can be particularly effective if you don’t carry out additional security checks.

Defence strategies against CWE-1007

To protect you from the threat of CWE-1007, technical and organisational measures are necessary. Here are some strategies that can help you minimise risk:

Unicode normalisation : One of the most effective methods of recognising homoglyphs is Unicode normalisation. Unicode normalisation converts similar-looking characters into standard forms, making identifying them more accessible. This can prevent different writing systems from being used to deceive.

javaCopy

importjava.text.Normalizer;   publicclassUnicodeNormalizationExample{       publicstaticvoidmain(String[]args){           StringsuspiciousString="päypäl.com";           StringnormalizedString=Normalizer.normalize(suspiciousString,Normalizer.Form.NFKC);           System.out.println("Normalized String: "+normalizedString);       }   }

This example shows how characters can be normalised to ensure that visually similar but different characters are recognised as such.

User training : Training in dealing with suspicious emails, links and domain names is an essential line of defence. You should learn to check URLs carefully and be aware that characters from different writing systems can be used to deceive you. This training should include regular exercises and examples to increase awareness and improve your ability to recognise such attacks.

Security warnings in the browser : Modern browsers have mechanisms to warn you if you visit a suspicious domain or use characters from different Unicode writing systems. These warnings should be enabled to protect you from such threats. Browser extension developers could develop additional filtering mechanisms that promptly alert you to possible deception.

Code reviews and static code analysis tools : In software projects, developers should conduct code reviews to identify suspicious characters. Static code analysis tools can also help detect homoglyphs in code and mitigate potential security risks.

javaCopy

publicclassCodeReviewExample{       publicstaticbooleancontainsSuspiciousCharacters(Stringinput){           // Checks whether the string contains non-Latin characters           return!input.matches("^[\ -\~]*$");       }       publicstaticvoidmain(String[]args){           Stringinput="paypaı.com";// contains the homoglyph "ı" (small i without a dot)           if(containsSuspiciousCharacters(input)){               System.out.println("Suspicious characters found: "+input);           }       }   }

This example shows how a simple check can help identify potentially dangerous signs and act accordingly.

Allowing specific character sets : Another risk mitigation measure is limiting the use of certain characters. For example, an application might specify that only Latin characters are allowed in usernames or URL paths to reduce the risk of homoglyph attacks. This restriction helps reduce the attack surface.

javaCopy

publicclassCharacterWhitelistExample{       publicstaticbooleanisValidInput(Stringinput){           returninput.matches("^[a-zA-Z0-9]*$");       }       publicstaticvoidmain(String[]args){           Stringusername="username";// contains a non-Latin character           if(isValidInput(username)){               System.out.println("Username is valid.");           }else{               System.out.println("Username contains invalid characters.");           }       }   }

This example shows a simple method for restricting the allowed characters in input to prevent using homoglyphs.

Technical challenges in homoglyph recognition

Recognising homoglyphs represents a significant technical challenge. One reason for this is the number of characters in the Unicode standard. Unicode includes thousands of characters from different writing systems that may look similar or identical. An algorithm intended to identify such characters must be able to distinguish between visual similarities and actual meaning.

Another problem is that not all applications or systems can display Unicode correctly. In some cases, different characters may appear identical through the rendering process, making it even more difficult for you to distinguish between legitimate and fake content. These technical challenges require the development of robust checking mechanisms that ensure that such attempts at deception can be detected.

Best practices for developers

As a developer, you play a crucial role in preventing CWE-1007. Here are some best practices to consider when developing secure applications:

Input validation : User input data should always be validated to ensure that no dangerous characters are used. If possible, only certain character sets should be allowed.

javaCopy

publicclassInputValidationExample{       publicstaticbooleanisValidInput(Stringinput){           returninput.matches("^[a-zA-Z0-9]*$");       }       publicstaticvoidmain(String[]args){           StringuserInput="hello123";           if(isValidInput(userInput)){               System.out.println("Input is valid.");           }else{               System.out.println("Input contains invalid characters.");           }       }   }

Escape and encode : Data used for display or transmission should always be escaped and encoded to ensure no harmful characters are inserted unnoticed.

xmlCopy

import org.apache.commons.text.StringEscapeUtils;   public class EscapeEncodeExample {       public static void main(String[] args) {           String userInput = "<script>alert('XSS');</script>";           String escapedInput = StringEscapeUtils.escapeHtml4(userInput);           System.out.println("Escaped Input: " + escapedInput);       }   }

This example shows how potentially malicious input can be correctly handled to prevent attacks such as Cross-Site Scripting (XSS).

Conscious use of fonts : Choosing the right font can help you recognise homoglyphs better. Some fonts distinguish more clearly between similar-looking characters, making it easier for you to spot differences. Examples of such fonts include ‘Consolas’, ‘Courier New’, and ‘DejaVu Sans Mono’. These fonts are handy when characters need to be clearly distinguished from each other, for example, in source code or security-related information.

Provide additional contextual information : If you’re asked to review sensitive information like URLs or usernames, it can be helpful to provide additional contextual information to help verify legitimacy. This includes warnings or visual markers that alert you that certain signs may be potentially dangerous.

Case Study: Homoglyph Domain Attack

A famous case study that illustrates the danger of homoglyphs is the attack via a fake “apple.com” domain. In this case, the attacker registered a domain that consisted of Cyrillic characters that visually looked identical to “apple.com.” When you clicked on this domain, you were taken to a website that looked almost exactly like the genuine Apple website. This allowed the attacker to steal sensitive information, such as login credentials.

Such attacks highlight the importance of paying attention to the visual distinction of characters when presenting information. Without careful examination, you would have no technical way of recognising the fake domain. Such attacks highlight the need to use technical and human defences to protect yourself from deception.

Homoglyph recognition tools

There are some tools and libraries designed explicitly for homoglyph recognition. These tools can help you, as a developer and security professional identify and remediate potential vulnerabilities in your systems. The most well-known tools include:

DNSTwist : A tool used to generate similar domain names and check whether they have been registered for phishing attacks. DNSTwist can help detect potential threats early and take countermeasures.

Homoglyph Detector Libraries : Various program libraries recognise homoglyphs in strings. These libraries can be integrated into applications to ensure suspicious characters are detected. Such libraries are particularly useful when users provide security-relevant input.

Unicode-Analyse-Tools : These tools can help check characters for their Unicode representation, ensuring that similar characters are not used unnoticed. Unicode analysis tools can help you identify and fix problems early.

Which CWEs are often used in conjunction with CWE-1007?

CWE-1007 is often used with other vulnerabilities that undermine the security and trustworthiness of applications. Here are some of the most common CWEs that frequently appear associated with CWE-1007:

CWE-601: Open Redirect:

This vulnerability occurs when an application allows users to be redirected to a different, potentially dangerous URL without sufficient validation. CWE-1007 can be used here to deceive users by presenting a seemingly legitimate but homoglyph-manipulated URL redirecting to a malicious resource.

CWE-643: Improper Neutralization of Data within XPath Expressions:

When user input is used in XPath queries without sufficient validation, CWE-1007 can allow attackers to use visually similar characters to manipulate the query and access data that would not usually be available.

CWE-79: Cross-Site Scripting (XSS):

Cross-site scripting occurs when an application does not adequately escape or validate user input returned for display. CWE-1007 can also be dangerous here if homoglyphs are used to hide malicious scripts in a context that looks harmless to the user.

CWE-20: Improper Input Validation:

This general vulnerability occurs when an application does not sufficiently validate user input. CWE-1007 is particularly problematic if no normalisation and validation measures are taken, as visually similar characters could pass as legitimate input.

CWE-77: Command Injection:

Command injection occurs when a user’s input is used in a way that results in the execution of system commands. By using homoglyphs, attackers could manipulate input so that the actual command deviates from the expected execution, which could give an attacker control over the system.

Combining CWE-1007 with these other vulnerabilities allows attackers to carry out significantly more complex and effective attacks. They leverage the visual deception of homoglyphs to exploit vulnerabilities that might be easier to detect without such deception.

Example application in Vaadin Flow

To illustrate all the best practices discussed in a real application, here is an example application in Vaadin Flow. Vaadin Flow is a popular open-source Java framework for building modern web applications. This sample application shows how you can implement Unicode normalisation, input validation, escaping, and encoding methods in a secure web application to prevent homoglyph-based attacks.

javaCopy

importcom.vaadin.flow.component.button.Button;importcom.vaadin.flow.component.notification.Notification;importcom.vaadin.flow.component.orderedlayout.VerticalLayout;importcom.vaadin.flow.component.textfield.TextField;importcom.vaadin.flow.router.Route;importorg.apache.commons.text.StringEscapeUtils;importjava.text.Normalizer;@Route("homoglyph-protection")publicclassHomoglyphProtectionViewextendsVerticalLayout{    publicHomoglyphProtectionView(){        // Username input field        TextFielduserInputField=newTextField("Enter username");        userInputField.setHelperText("Only Latin letters and numbers allowed.");        // Validation button        ButtonvalidateButton=newButton("Validate",event->{            StringuserInput=userInputField.getValue();            // Step 1: Unicode normalization            StringnormalizedInput=Normalizer.normalize(userInput,Normalizer.Form.NFKC);            // Step 2: Input validation - only Latin letters and numbers            if(!isValidInput(normalizedInput)){                Notification.show("Username contains invalid characters.");                return;            }            // Step 3: Escape potentially harmful input            StringescapedInput=StringEscapeUtils.escapeHtml4(normalizedInput);            // Notification if input is valid            Notification.show("Input is valid: "+escapedInput);        });        add(userInputField,validateButton);    }    // Input validation method that only allows Latin letters and numbers    privatebooleaninvalidinput(Stringinput){        returninput.matches("^[a-zA-Z0-9]*$");    }}

In this Vaadin Flow application, what you have learned before is put into practice:

Unicode normalisation : User input is normalised to ensure that visually similar characters that have different Unicode values ​​are treated correctly.

Input validation : The input is checked to allow only Latin letters and numbers. This prevents the use of homoglyphs from other character sets.

Escape and encode : The input is escaped before further processing to remove potentially harmful characters and prevent attacks such as Cross-Site Scripting (XSS).

This sample application shows how to apply the secure user input best practices discussed previously in a real-world web application to minimise homoglyph risks.

Conclusion

CWE-1007, insufficient visual discrimination of homoglyphs, is a vulnerability that is becoming increasingly important in today’s digital world. Attackers use the visual similarity of characters to deceive you, steal your data, or compromise systems. The threat of homoglyph attacks can have serious consequences, especially if you and your systems are not adequately trained and prepared.

Preventing such attacks requires technical solutions, user training, and developer best practices. By implementing appropriate defence strategies, you can minimise the risk of homoglyph attacks and ensure the security of your systems and data. Everyone must be involved—from developers to end users—be aware of the risks and take steps to mitigate them.

Next Steps

To better understand and protect yourself from CWE-1007, you should:

• Train your employees about the danger of homoglyphs and teach them to recognize suspicious characters.
• Enable security features in your browsers and applications to warn users about fake domains.
• Integrate homoglyph recognition tools and libraries into your software development processes.
• Conduct regular security scans and code reviews to ensure no malicious characters have been inserted into your systems.

Security is an ongoing process, and the homoglyph threat requires continued vigilance and adaptation. Stay informed, train your team, and use the right tools to ensure homoglyphs aren’t a gap in your security strategy. Together, we can help improve the security of our digital world and successfully ward off homoglyph threats.

Happy Coding

Sven

]]>JavaSecure Coding PracticesUncategorizedVaadinhttps://svenruppert.com/posts/the-history-of-parallel-processing-in-java-from-threads-to-virtual-threads/Fri, 01 Nov 2024 22:02:37 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/the-history-of-parallel-processing-in-java-from-threads-to-virtual-threads/Since the early days of computer science, parallel processing has represented one of the greatest challenges and opportunities. Since its inception in 1995, Java has undergone a significant journey in the world of parallel programming to provide developers with ever-better tools. This story is a fascinating journey through threads, the executor framework, fork/join, parallel streams, CompletableFuture and the latest developments in Project Loom. In this blog post, we take a detailed look at the evolution of parallel processing in Java and the innovations that came with it.<![CDATA[

Since the early days of computer science, parallel processing has represented one of the greatest challenges and opportunities. Since its inception in 1995, Java has undergone a significant journey in the world of parallel programming to provide developers with ever-better tools. This story is a fascinating journey through threads, the executor framework, fork/join, parallel streams, CompletableFuture and the latest developments in Project Loom. In this blog post, we take a detailed look at the evolution of parallel processing in Java and the innovations that came with it.

1. Locking mechanisms in thread programming in Java
   1. synchronized keyword
   2. ReentrantLock
   3. ReadWriteLock
   4. StampedLock
   5. Semaphore
   6. Summary of the locking mechanisms
2. The early years: Threads andRunnable
   1. Using threads: advantages and disadvantages
3. Java 5: The Executor Framework andjava.util.concurrent
4. Java 7: Fork/Join Framework
5. Java 8: Parallel Streams
6. Java 8: CompletableFuture
7. Java 9 to Java 19: Improvements and Project Loom
8. The Evolution of Parallel Processing in Java: Conclusion
9. Looking into the future

Locking mechanisms in thread programming in Java

To make parallel processing safe and efficient, Java provides various locking mechanisms that help coordinate access to shared resources and avoid data corruption. These mechanisms are critical to ensure data consistency and integrity, especially when multiple threads access the same resources simultaneously. The most critical locking mechanisms in Java are described below:

synchronized keyword

Thesynchronized keyword is the most basic mechanism for synchronising threads in Java. It ensures that only one thread can access a synchronised method or code block at a time. This mechanism uses a so-called monitor, which is automatically locked and unlocked.

javaCopy

publicsynchronizedvoidincrement(){    // Critical section    counter++;}

Advantages:

- Easy to use and integrate into code.

- Automatic management of bans.

Disadvantages:

- Can lead to performance issues if many threads want to access the synchronised resource simultaneously.

- No flexibility, such as B. the ability to set timeouts or conditional locks.

ReentrantLock

ReentrantLock is a class from thejava.util.concurrent.locks package that provides greater flexibility than thesynchronized keyword. As the name suggests,ReentrantLock is a “reentrant” lock, meaning that a thread that already holds the lock can reacquire it without getting into a deadlock.

javaCopy

privatefinalReentrantLocklock=newReentrantLock();publicvoidincrement(){    lock.lock();    try{        // Critical section        counter++;    }finally{        lock.unlock();    }}

Advantages:

- Provides more control over the locking process, e.g. B. the ability to acquire locks with timeouts (tryLock).

- Supports fair locking, which ensures that threads are granted access in the requested order (lock(true)).

Disadvantages:

- Requires manual release of locks, increasing the risk of deadlocks ifunlock is not performed correctly.

ReadWriteLock

ReadWriteLock is a unique locking mechanism that distinguishes read and write locks. It consists of two locks: a read lock and a write lock. Multiple threads can acquire a read lock simultaneously as long as no write operation is performed. However, only one thread can acquire the write lock, keeping writes exclusive.

javaCopy

privatefinalReentrantReadWriteLocklock=newReentrantReadWriteLock();publicvoidread(){    lock.readLock().lock();    try{        // Read operation    }finally{        lock.readLock().unlock();    }}publicvoidwrite(){    lock.writeLock().lock();    try{        // Write operation    }finally{        lock.writeLock().unlock();    }}

Advantages:

- Increases concurrency as multiple threads can read simultaneously as long as there are no writes.

- Reduces the likelihood of blocking read access.

Disadvantages:

- More complex to use thansynchronized orReentrantLock.

- Requires careful design to ensure no deadlocks or race conditions occur.

StampedLock

StampedLock is another variant ofReadWriteLock introduced in Java 8. UnlikeReadWriteLock,StampedLock provides optimistic read locks that enable even higher concurrency. A stamp is returned each time the lock is acquired to ensure the data remains valid.

javaCopy

privatefinalStampedLocklock=newStampedLock();publicvoidread(){    longstamp=lock.tryOptimisticRead();    // Read operation    if(!lock.validate(stamp)){        stamp=lock.readLock();        try{            // Read again        }finally{            lock.unlockRead(stamp);        }    }}

Advantages:

- Provides optimistic read locks that enable high concurrency as long as no writes occur.

- Lower read operation overhead compared to traditional locks.

Disadvantages:

- Complex to use as developers need to ensure that the stamp is valid.

- No “re-enterable” locks, which may limit usage for some scenarios.

Semaphore

Semaphore is another synchronisation mechanism that allows several threads to access a resource simultaneously. It is often used to control simultaneous access to limited resources.

javaCopy

privatefinalSemaphoresemaphore=newSemaphore(3);publicvoidaccessResource(){    try{        semaphore.acquire();        // Access the resource    }catch(InterruptedExceptione){        Thread.currentThread().interrupt();    }finally{        semaphore.release();    }}

Advantages:

- Allows you to limit the number of threads that are allowed to access a resource at the same time.

- Flexible and applicable to various scenarios, e.g. B. to implement pooling mechanisms.

Disadvantages:

- Can become complex when using multiple semaphores in an application.

- Requires careful management to ensure thatacquire andrelease are called correctly.

Summary of the locking mechanisms

Java offers a variety of locking mechanisms, each suitable for different scenarios. Thesynchronized keyword is easy to use, but less flexible for complex scenarios.ReentrantLock andReadWriteLock provide more control and enable higher parallelism, whileStampedLock is suitable for particularly demanding read operations.Semaphore, on the other hand, is ideal for controlling concurrent access to limited resources. Choosing the proper mechanism depends on the application’s requirements, particularly regarding concurrency, resource contention, and maintainability.

The early years: Threads andRunnable

Initially, Java’s concept of threads was the cornerstone of parallel processing. Java was developed as a platform-independent language intended for networked, distributed systems. With a focus on multithreading, Java offered a built-inThread class and theRunnable interface, making it possible to execute multiple tasks simultaneously.

javaCopy

classMyTaskimplementsRunnable{    @Override    publicvoidrun(){        // Task running in parallel        System.out.println("Hello from thread: "+Thread.currentThread().getName());    }}publicclassMain{    publicstaticvoidmain(String[]args){        Threadthread=newThread(newMyTask());        thread.start();    }}

In the early versions of Java, threads were the only option for parallel processing. These were supported by operating system threads, which meant managing threads could be heavy and resource-intensive. Developers had to take care of details such as synchronizing shared resources and avoiding deadlocks themselves, which made developing parallel applications challenging.

Using threads: advantages and disadvantages

Advantages of threads

Simple modelling of parallel tasks : Threads allow developers to divide parallel tasks into separate units that can run concurrently.

Direct control : Threads give developers fine control over parallel execution, which is particularly useful when specific thread management requirements exist.

Good support from the operating system : Threads are supported directly by the operating system, meaning they can access all system resources and benefit from operating system optimisations.

Disadvantages of threads

Complexity of synchronisation : When multiple threads access shared resources, developers must use synchronisation mechanisms such assynchronized blocks orlocks to avoid data corruption. Such mechanisms ensure that only one thread can access a critical resource at a time, ensuring data consistency. However, this often results in complicated and error-prone code, as developers must carefully ensure that all necessary sections are correctly synchronised. An incorrectly set or forgotten synchronisation block can lead to severe errors, such as race conditions, in which the program’s output depends on the timing of thread executions. Additionally, synchronisation mechanisms such aslocks orsynchronized blocks can lead to a performance penalty as threads often have to wait until a resource is released, which limits parallelism. These wait times can cause bottlenecks in more complex applications, mainly when multiple threads compete for different resources. Therefore, correctly applying synchronisation techniques requires a deep understanding of thread interactions and careful design to ensure data integrity and maximise performance.

Resource intensive : Each thread requires space for its stack and additional resources such as thread handling and context switching. These resource requirements can quickly add up with many threads and lead to system overload. In particular, the memory consumption for the thread stacks and the management of the threads by the operating system lead to increased resource requirements. With a large number of threads, the frequency of context switches also increases, which can lead to a significant reduction in overall performance. This often makes threads inefficient and difficult to manage at a large scale.

The danger of deadlocks : When using synchronisation mechanisms, there is a risk of deadlocks when two or more threads are blocked in a cyclic dependency on each other. Deadlocks often arise when multiple threads hold different locks in a mismatched order, waiting for other threads to release the needed resources. This leads to a situation where none of the threads can continue working because they are all waiting for the others to release resources. Deadlocks are often difficult to reproduce and debug because they only occur under certain runtime conditions. Strategies to avoid deadlocks include using timeout mechanisms, avoiding nested locks, and implementing lock ordering schemes to ensure all threads acquire locks in the same order.

Difficult scalability : Manual management of threads makes it difficult to scale applications, especially on systems with many processor cores. One of the main reasons for this is the challenge of determining the optimal number of threads to use system resources efficiently. Too many threads can cause system overload because management and context switching between threads consume significant CPU resources. On the other hand, too few threads can result in under-utilization of available processor resources, degrading overall performance. Additionally, it is challenging to adapt thread management to the dynamic needs of an application, especially when the load is variable. Developers often have to resort to complex heuristics or dynamic thread pools to control the number of active threads, which significantly complicates the implementation and maintenance of the application. These challenges make it complicated to efficiently scale applications to modern multicore processors because the balance between parallelism and overhead is challenging.

Java 5: The Executor Framework andjava.util.concurrent

The introduction of Java 5 in 2004 introduced thejava.util.concurrent package, intended to address many of the problems of early parallel programming in Java. The Executor framework enabled higher abstraction of threads. Instead of starting and managing threads manually, developers could now rely on a task-based architecture to pass tasks to anExecutorService.

The Executor framework introduced classes likeThreadPoolExecutor,ScheduledExecutorService, and many synchronization helper classes likeSemaphore,CountDownLatch, andConcurrentHashMap. This not only made thread management easier but also led to more efficient use of system resources.

javaCopy

ExecutorServiceexecutor=Executors.newFixedThreadPool(4);executor.submit(()->{    System.out.println("Task is running in thread: "+Thread.currentThread().getName());});executor.shutdown();

The Executor Framework changed the way developers modelled parallel tasks. Instead of focusing on thread creation, they could define tasks and let the infrastructure handle execution.

Java 7: Fork/Join Framework

Java 7 introduced the Fork/Join framework in 2011, explicitly designed for computationally intensive tasks that could be broken down into smaller subtasks. The fork/join framework provided a powerful recursion and divide-and-conquer infrastructure, allowing complex problems to be broken down into smaller, more manageable sub-problems. This division enabled the efficient use of modern multi-core processors.

The fork/join framework was particularly useful for problems broken down into independent subproblems, such as B. calculating Fibonacci numbers, sorting large arrays (e.g. with merge sort) or processing large amounts of data in parallel. The central component of the framework is theForkJoinPool, which handles the management of task transfer between threads. TheForkJoinPool uses the so-called work stealing process, in which less busy threads take over work from more busy threads. This ensures better load balancing and increases the efficiency of parallel processing.

Another advantage of the fork/join framework is the ability to handle recursive tasks efficiently. Developers can use classes likeRecursiveTask orRecursiveAction to define tasks that either provide a return value (RecursiveTask) or do not require a return (RecursiveAction). The fork/join approach makes it possible to recursively split the tasks (fork) and then combine the results again (join).

xmlCopy

ForkJoinPool forkJoinPool = new ForkJoinPool();forkJoinPool.invoke(new RecursiveTask<Integer>() {    @Override    protected Integer compute() {        // Split and calculate task        if (/* base case reached */) {            return 42;  // Example return value        } else {            // Divide the task further            RecursiveTask<Integer> subtask1 = new Subtask();            RecursiveTask<Integer> subtask2 = new Subtask();            subtask1.fork();            subtask2.fork();            return subtask1.join() + subtask2.join();        }    }});

The fork/join framework delivered significant performance improvements for CPU-intensive workloads, particularly for tasks easily broken down into smaller, independent pieces. It made it easier to write parallel algorithms without the developer worrying about thread distribution details. The ‘ForkJoinPool’ handles the distribution of tasks and uses work stealing to ensure that the processor resources are used optimally. This significantly increases performance compared to manual thread management, especially for computationally intensive and highly parallelisable tasks.

Java 8: Parallel Streams

Java 8, released in 2014, marked a milestone in the evolution of Java, particularly with the introduction of lambda expressions, streams, and functional programming. These new features made the language more flexible and easier to use, especially for parallel operations. One of the most significant new features for parallel processing was Parallel Streams.

Parallel Streams allowed developers to effortlessly parallelise operations across collections without explicitly dealing with threads or synchronisation. This was achieved by integrating the fork/join framework behind the scenes. Parallel streams use theForkJoinPool internally to distribute tasks efficiently across multiple processor cores. This approach is based on the divide-and-conquer design principle, in which an enormous task is broken down into smaller subtasks that can be executed in parallel.

The developer uses theparallelStream() method to convert a collection into a parallel stream. This results in the processing of the collection’s elements occurring simultaneously, with individual parts of the task being automatically distributed across the available CPU cores. In contrast to manual thread management, this approach offers a high level of abstraction and relieves the developer of the complex management of threads and synchronisation.

An example of using Parallel Streams is processing a list of numbers in parallel. Here, the operations applied to each element are carried out simultaneously, which can achieve a significant increase in performance on large data sets:

xmlCopy

List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5);numbers.parallelStream().map(n -> n * n).forEach(System.out::println);

The design of Parallel Streams aims to simplify the development of parallel applications by using a declarative syntax that allows the developer to focus on the logic of data processing rather than on the details of thread management and distribution of the tasks to deal with. This higher abstraction makes parallel processing more accessible, which leads to better performance, especially in multi-core systems.

Java 8: CompletableFuture

TheCompletableFuture API was also introduced in Java 8 and significantly expanded the possibilities of asynchronous programming.CompletableFuture allows the creation, chaining and combining asynchronous tasks, making it a handy tool for developing event-driven applications. Using methods likethenApply,thenAccept andthenCombine makes it easy to define a sequence of asynchronous operations that should be executed sequentially or whose results can be combined.

ACompletableFuture represents a future calculation and allows the different steps of the workflow to be defined declaratively. For example, an asynchronous calculation can be started, and the result can be processed further without worrying about explicitly managing the threads. This significantly simplifies the code and makes it more readable.

An example of usingCompletableFuture shows how to execute multiple asynchronous operations one after the other:

javaCopy

CompletableFuture.supplyAsync(()->"Hello")                .thenApply(s->s+" World")                .thenAccept(System.out::println);

In this example, an asynchronous calculation is first started that returns the string “Hello”. Then, the result of this calculation is modified by thethenApply method by adding " World". Finally,thenAccept prints the result on the console. The entire process occurs asynchronously, without the developer worrying about explicitly managing threads.

The architecture ofCompletableFuture is based on the concept of so-called “completion stages”, which allow the creation of asynchronous pipelines. EachCompletion Stage can either trigger a new calculation or process the result of a previous calculation. This enables the modelling of complex workflows, such as: E.g. executing multiple tasks in parallel and then combining the results (thenCombine), or defining actions that should be carried out in the event of an error (exceptionally).

Another significant advantage ofCompletableFuture is the ability to combine asynchronous tasks. For example, two independent asynchronous calculations can be performed in parallel, and their results can then be merged:

xmlCopy

CompletableFuture<Integer> future1 = CompletableFuture.supplyAsync(() -> 10);CompletableFuture<Integer> future2 = CompletableFuture.supplyAsync(() -> 20);CompletableFuture<Integer> combinedFuture = future1.thenCombine(future2, (result1, result2) -> result1 + result2);combinedFuture.thenAccept(result -> System.out.println("Combined Result: " + result));

In this example, two calculations are performed in parallel, and their results are combined. ThethenCombine method allows the results of the two futures to be added, andthenAccept prints the combined result.

This model allows complex workflows to be created without explicitly relying on threads or callbacks, making the code cleaner, more modular, and easier to maintain.CompletableFuture also provides methods such asallOf() andanyOf() that allow multiple futures to be monitored and processed simultaneously. This is particularly useful for scenarios where numerous independent tasks must be executed in parallel.

Overall, theCompletableFuture API makes asynchronous programming in Java much more accessible and allows developers to develop reactive and non-blocking applications with relatively little effort.

Java 9 to Java 19: Improvements and Project Loom

After Java 8, the parallel programming models continuously improved. Java 9 brought improvements to the fork/join framework and introduced the ‘Flow’ API, which supported a reactive streaming model. Java 9 to 17 focused primarily on performance improvements, security, and the modularization of the JDK (Project Jigsaw).

However, one of the most significant innovations in recent times is Project Loom. Since Java 19,Virtual Threads has been available as a preview feature to revolutionise parallel programming in Java. Virtual threads are lightweight threads that enable many concurrent tasks to run without the typical overhead of traditional operating system threads. While traditional threads are limited by resource costs (such as memory for stacks and context switches), virtual threads are designed to be much more resource-efficient. This means developers can create millions of virtual threads that work independently without overloading the system.

Virtual threads are handy for server-side applications that handle many concurrent connections, such as web servers or microservices. In traditional approaches, handling each request in its thread often leads to scaling problems because hardware resources limit the number of possible threads. Virtual threads, on the other hand, allow each incoming request to be handled in its virtual thread, significantly increasing parallelism.

Virtual threads work because they are efficiently managed by the Java runtime system rather than directly by the operating system like traditional threads. This significantly speeds up context switching and makes managing millions of threads realistic. Virtual threads are programmed like traditional threads, allowing existing code to be easily adapted to take advantage of this new technology.

A simple example of using virtual threads shows how to use anexecutor to execute a task in a virtual thread:

javaCopy

try(varexecutor=Executors.newVirtualThreadPerTaskExecutor()){    executor.submit(()->{        System.out.println("Running in a virtual thread");    });}

This example creates an executor that uses a new virtual thread for each task. Thesubmit method starts the task in a virtual thread, which requires significantly fewer resources than a traditional thread.

Project Loom can potentially make parallel programming in Java much more accessible by eliminating the need for developers to worry about thread scaling. Virtual threads are significantly more efficient and offer much higher parallelism without the programmer having to work with thread pools or complex synchronisation mechanisms explicitly. This increased scalability is particularly valuable in applications where concurrent operations must be dynamically scaled, as in many modern web applications and cloud environments. The introduction of virtual threads makes Java an even stronger choice for developing highly scalable, parallel applications by dramatically reducing the complexity of thread management.

The Evolution of Parallel Processing in Java: Conclusion

The journey of parallel processing in Java reflects the language’s evolutionary nature. From the early days of threads, when developers had to rely on low-level APIs, to the highly abstract paradigms such as the Executor framework, Fork/Join, and Parallel Streams, Java has continually introduced improvements to make parallel application development easier.

With recent developments such as ‘CompletableFuture’ and Project Loom, Java can meet the needs of modern software development, especially in scalability and performance. Parallel processing in Java is now simpler, safer and more efficient than ever before - providing developers with powerful tools to exploit the full potential of modern multicore systems.

Looking into the future

With Project Loom on the path to stability, we could see a further shift in focus towards even simpler and more performant parallel processing techniques. Virtual threads will likely pave the way for new frameworks and libraries that benefit from this lightweight parallelism. Developers will continue to have access to the best parallel processing tools without worrying about thread management’s complexities.

Java has proven that it can keep changing with the times as a parallel programming language—a language that rises to the challenges and adapts to the needs of modern developers. Java’s history of parallel processing is one of progress and continuous innovation. With the upcoming developments, it will remain exciting to see what new possibilities the future will bring us.

]]>ConcurrencyJavahttps://svenruppert.com/posts/cwe-778-lack-of-control-over-error-reporting-in-java/Fri, 18 Oct 2024 14:07:20 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cwe-778-lack-of-control-over-error-reporting-in-java/Learn how inadequate control over error reporting leads to security vulnerabilities and how to prevent them in Java applications. Safely handling error reports is a central aspect of software development, especially in safety-critical applications. CWE-778 describes a vulnerability caused by inadequate control over error reports. This post will analyse the risks associated with CWE-778 and show how developers can implement safe error-handling practices to avoid such vulnerabilities in Java programs.<![CDATA[

Learn how inadequate control over error reporting leads to security vulnerabilities and how to prevent them in Java applications.

Safely handling error reports is a central aspect of software development, especially in safety-critical applications. CWE-778 describes a vulnerability caused by inadequate control over error reports. This post will analyse the risks associated with CWE-778 and show how developers can implement safe error-handling practices to avoid such vulnerabilities in Java programs.

1. Learn how inadequate control over error reporting leads to security vulnerabilities and how to prevent them in Java applications.
2. What is CWE-778?
   1. Examples of CWE-778 in Java
3. Secure error handling
   1. Example with Vaadin Flow
4. Using design patterns to reuse logging and error handling.
   1. Decorator Pattern
   2. Proxy Pattern
   3. Template Method Pattern
5. Evaluate log messages for attack detection in real time
6. Best practices for avoiding CWE-778
7. Conclusion

What is CWE-778?

The Common Weakness Enumeration (CWE) defines CWE-778 as a vulnerability where bug reporting is inadequately controlled. Bug reports often contain valuable information about an application’s internal state, including system paths, configuration details, and other sensitive information that attackers can use to identify and exploit vulnerabilities. Improper handling of error reports can result in unauthorised users gaining valuable insight into the application’s system structure and logic.

Exposing such information in a security-sensitive application could have potentially serious consequences, such as the abuse of SQL injection or cross-site scripting (XSS) vulnerabilities. Therefore, it is critical that bug reports are carefully controlled and only accessible to authorised individuals.

Examples of CWE-778 in Java

The following example considers a simple Java application used to authenticate users:

javaCopy

publicclassUserLogin{    publicstaticvoidmain(String[]args){        try{            authenticateUser("admin","wrongpassword");        }catch(Exceptione){            // Error is output directly to the user            System.out.println("Error: "+e.getMessage());            e.printStackTrace();        }    }    privatestaticvoidauthenticateUser(Stringusername,Stringpassword)throwsException{        if(!"correctpassword".equals(password)){            thrownewException("Invalid password for user: "+username);        }    }}

This example displays an error message if the user enters an incorrect password. However, this approach has serious security gaps:

1. The error message contains specific information about the username.

2. The full stack trace is output, allowing an attacker to obtain details about the application’s implementation.

This information can help an attacker understand the application’s internal structure and make it easier for them to search specifically for additional vulnerabilities.

Secure error handling

To minimise the risks described above, secure error handling should be implemented. Instead of providing detailed information about the error, the user should only be shown a general error message:

javaCopy

publicclassUserLogin{    publicstaticvoidmain(String[]args){        try{            authenticateUser("admin","wrongpassword");        }catch(Exceptione){            // Generic error message to the user            System.out.println("Authentication failed. Please check your entries.");            // Logging the error in the log file (for admins)            logError(e);        }    }    privatestaticvoidauthenticateUser(Stringusername,Stringpassword)throwsException{        if(!"correctpassword".equals(password)){            thrownewException("Invalid password for user: "+username);        }    }    privatestaticvoidlogError(Exceptione){        // Error is securely logged without displaying it to the user        System.err.println("An error has occurred: "+e.getMessage());    }}

In this improved version, only a general error message is displayed to the user while the error is logged internally. This prevents sensitive information from being shared with unauthorised users.

Such errors should be logged in a log file accessible only to authorised persons. A logging framework such as Log4j or SLF4J provides additional mechanisms to ensure logging security and store only necessary information.

Example with Vaadin Flow

Vaadin Flow is a Java framework for building modern web applications, and CWE-778 can also be a problem if error reports are mishandled. A safe example of error handling in a Vaadin application could look like this:

javaCopy

importcom.vaadin.flow.component.button.Button;importcom.vaadin.flow.component.notification.Notification;importcom.vaadin.flow.component.textfield.PasswordField;importcom.vaadin.flow.component.textfield.TextField;importcom.vaadin.flow.router.Route;@Route("login")publicclassLoginViewextendsVerticalLayout{    publicLoginView(){        TextFieldusernameField=newTextField("Benutzername");        PasswordFieldpasswordField=newPasswordField("Password");        ButtonloginButton=newButton("Login",event->{            try{                authenticateUser(usernameField.getValue(),passwordField.getValue());            }catch(Exceptione){                // Generic error message to the user                Notification.show("Authentication failed. Please check your entries.");                // Logging the error in the log file (for admins)                logError(e);            }        });        add(usernameField,passwordField,loginButton);    }    privatevoidauthenticateUser(Stringusername,Stringpassword)throwsException{        if(!"correctpassword".equals(password)){            thrownewException("Invalid password for user: "+username);        }    }    privatevoidlogError(Exceptione){        // Error is securely logged without displaying it to the user        System.err.println("An error has occurred: "+e.getMessage());    }}

ThelogError method ensures that errors are logged securely without sensitive information being visible to the end user. Vaadin Flow enables the integration of such secure practices to ensure that bug reports are not leaked uncontrollably.

Using design patterns to reuse logging and error handling.

To promote the reuse of error handling and logging, design patterns that enable the modularization and unification of such tasks can be used. Two suitable patterns are theDecorator Pattern and theTemplate Method Pattern.

Decorator Pattern

The Decorator Pattern is a structural design pattern that allows an object’s functionality to be dynamically extended without changing the underlying class. This is particularly useful when adding additional responsibilities, such as logging, security checks, or error handling, without modifying the original class’s code.

The Decorator Pattern works by using so-called “wrappers”. Instead of modifying the class directly, the object is wrapped in another class that implements the same interface and adds additional functionality. In this way, different decorators can be combined to create a flexible and expandable structure.

A vital feature of the Decorator Pattern is its adherence to the open-closed principle, one of the fundamental principles of object-oriented design. The open-closed principle states that a software component should be open to extensions but closed to modifications. The Decorator Pattern does just that by allowing classes to gain new functionality without changing their source code.

In the context of error handling and logging, developers can write an introductory class for authentication, while separate decorators handle error logging and handling of specific errors. This leads to a clear separation of responsibilities, significantly improving code maintainability.

The following example shows the implementation of the Decorator pattern to reuse error handling and logging:

javaCopy

publicinterfaceAuthenticator{    voidauthenticate(Stringusername,Stringpassword)throwsException;}publicclassBasicAuthenticatorimplementsAuthenticator{    @Override    publicvoidauthenticate(Stringusername,Stringpassword)throwsException{        if(!"correctpassword".equals(password)){            thrownewException("Invalid password for user: "+username);        }    }}publicclassLoggingAuthenticatorDecoratorimplementsAuthenticator{    privatefinalAuthenticatorwrapped;    publicLoggingAuthenticatorDecorator(Authenticatorwrapped){        this.wrapped=wrapped;    }    @Override    publicvoidauthenticate(Stringusername,Stringpassword)throwsException{        try{            wrapped.authenticate(username,password);        }catch(Exceptione){            logError(e);            throwe;        }    }    privatevoidlogError(Exceptione){        System.err.println("An error has occurred: "+e.getMessage());    }}

In this example,**BasicAuthenticator** is used as the primary authentication class, while the**LoggingAuthenticatorDecorator** Added additional functionality, namely error logging. This decorator wraps the original authentication class and extends its behaviour. This allows the logic to be flexibly extended by adding more decorators, such as a**SecurityCheckDecorator**, which performs additional security checks before authentication.

An advantage of this approach is combining decorators in any order to achieve tailored functionality. For example, one could first add a security decoration and then implement error logging without changing the original authentication logic. This results in a flexible and reusable structure that is particularly useful in large projects where different aspects such as logging, security checks, and error handling are required in various combinations.

The Decorator Pattern is, therefore, a powerful tool for increasing software modularity and extensibility. It avoids code duplication, promotes reusability, and enables a clean separation of core logic and additional functionalities. This makes it particularly useful in secure error handling and implementing cross-cutting concerns such as logging in safety-critical applications.

The Decorator Pattern can add functionality, such as logging or error handling, to existing methods without modifying their original code. The following example shows how the Decorator Pattern enables centralised error handling:

javaCopy

publicinterfaceAuthenticator{    voidauthenticate(Stringusername,Stringpassword)throwsException;}publicclassBasicAuthenticatorimplementsAuthenticator{    @Override    publicvoidauthenticate(Stringusername,Stringpassword)throwsException{        if(!"correctpassword".equals(password)){            thrownewException("Invalid password for user: "+username);        }    }}publicclassLoggingAuthenticatorDecoratorimplementsAuthenticator{    privatefinalAuthenticatorwrapped;    publicLoggingAuthenticatorDecorator(Authenticatorwrapped){        this.wrapped=wrapped;    }    @Override    publicvoidauthenticate(Stringusername,Stringpassword)throwsException{        try{            wrapped.authenticate(username,password);        }catch(Exceptione){            logError(e);            throwe;        }    }    privatevoidlogError(Exceptione){        System.err.println("An error has occurred: "+e.getMessage());    }}

In this example, the**LoggingAuthenticatorDecorator** is a decorator for the class**BasicAuthenticator**. The Decorator Pattern allows error handling and logging to be centralised without changing the underlying authentication class.

Proxy Pattern

The proxy pattern is a structural design pattern used to control access to an object. It is often used to add functionality such as caching, access control, or logging. In contrast to the decorator pattern, primarily used to extend functionality, the proxy serves as a proxy that takes control of access to the actual object.

The proxy pattern ensures that all access to the original object occurs via the proxy, meaning specific actions can be carried out automatically. For example, the Proxy Pattern could ensure that authorised users can only access a particular resource while logging all accesses.

A typical example of the proxy pattern for encapsulating logging and error handling looks like this:

javaCopy

publicinterfaceAuthenticator{    voidauthenticate(Stringusername,Stringpassword)throwsException;}publicclassBasicAuthenticatorimplementsAuthenticator{    @Override    publicvoidauthenticate(Stringusername,Stringpassword)throwsException{        if(!"correctpassword".equals(password)){            thrownewException("Invalid password for user: "+username);        }    }}publicclassProxyAuthenticatorimplementsAuthenticator{    privatefinalAuthenticatorrealAuthenticator;    publicProxyAuthenticator(AuthenticatorrealAuthenticator){        this.realAuthenticator=realAuthenticator;    }    @Override    publicvoidauthenticate(Stringusername,Stringpassword)throwsException{        logAccessAttempt(username);        try{            realAuthenticator.authenticate(username,password);        }catch(Exceptione){            logError(e);            throwe;        }    }    privatevoidlogAccessAttempt(Stringusername){        System.out.println("Authentication attempt for user: "+username);    }    privatevoidlogError(Exceptione){        System.err.println("An error has occurred: "+e.getMessage());    }}

In this example,BasicAuthenticator is wrapped by a**ProxyAuthenticator**, which controls all calls to theauthenticate method. The proxy adds additional functionality, such as access and error logging, ensuring that all access goes through the proxy before the authentication object is called.

A key difference between the Proxy Pattern and the Decorator Pattern is that the Proxy primarily controls access to the object and its use. The proxy can check access rights, add caching, or manage an object’s lifetime. The Decorator Pattern, on the other hand, is designed to extend an object’s behaviour by adding additional responsibilities without changing the access logic.

In other words, the Proxy Pattern acts as a protection or control mechanism, while the Decorator Pattern adds additional functionality to extend the behaviour. Both patterns are very useful when integrating cross-cutting concerns such as logging or security checks into the application, but they differ in their focus and application.

Template Method Pattern

The Template Method Pattern allows for defining the general flow of a process while implementing specific steps in subclasses. This ensures that error handling remains consistent:

javaCopy

publicabstractclassAbstractAuthenticator{    publicfinalvoidauthenticate(Stringusername,Stringpassword){        try{            doAuthenticate(username,password);        }catch(Exceptione){            logError(e);            thrownewRuntimeException("Authentication failed. Please check your entries.");        }    }    protectedabstractvoiddoAuthenticate(Stringusername,Stringpassword)throwsException;    privatevoidlogError(Exceptione){        System.err.println("An error has occurred: "+e.getMessage());    }}publicclassConcreteAuthenticatorextendsAbstractAuthenticator{    @Override    protectedvoiddoAuthenticate(Stringusername,Stringpassword)throwsException{        if(!"correctpassword".equals(password)){            thrownewException("Invalid password for user: "+username);        }    }}

The Template Method Pattern centralises error handling in the**AbstractAuthenticator** class so that all subclasses use the same consistent error handling strategy.

Evaluate log messages for attack detection in real time

Another aspect of secure error handling is using log messages to detect attacks in real-time. Analysing the log data can identify potential attacks early, and appropriate measures can be taken. The following approaches are helpful:

Centralised logging : Use a central logging platform like the ELK Stack (Elasticsearch, Logstash, Kibana) or Splunk to collect all log data in one place. This enables comprehensive analysis and monitoring of security-related incidents.

Pattern recognition : Create rules and patterns that identify potentially malicious activity, such as multiple failed login attempts in a short period. Such rules can trigger automated alerts when suspicious activity is detected.

Anomaly detection : Machine learning techniques detect anomalous activity in log data. A sudden increase in certain error messages or unusual access patterns could indicate an ongoing attack.

Real-time alerts : Configure the system so that certain security-related events trigger alerts in real-time. This allows administrators to respond immediately to potential threats.

Analyse threat intelligence : Use log messages to collect and analyse threat intelligence. For example, IP addresses that repeatedly engage in suspicious activity can be identified, and appropriate action can be taken, such as blocking the address.

Integration into SIEM systems : Use security information and event management (SIEM) systems to correlate log data from different sources and gain deeper insights into potential threats. SIEM systems often also provide tools to automate responses to specific events.

By combining these approaches, attacks can be detected early, and the necessary steps can be taken to limit the damage.

Best practices for avoiding CWE-778

To avoid CWE-778 in your applications, the following best practices should be followed:

Generic error messages : Avoid sharing detailed information about errors with end users. Error messages should be worded as generally as possible to avoid providing clues about the internal implementation.

Error logging : Use logging frameworks like Log4j or SLF4J to log errors securely. This allows bugs to be tracked internally without exposing sensitive information.

No stack traces to users : Make sure stack traces are only visible in the log and are not output to the user. Instead, generic error messages that do not contain technical details should be used.

Access control : Ensure that only authorized users have access to detailed error reports. Error logs should be well-secured and viewable only by administrators or developers.

Regular error testing and security analysis : Run regular tests to ensure that error handling works correctly. Static code analysis tools help detect vulnerabilities like CWE-778 early.

Avoiding sensitive information : To prevent sensitive information such as usernames, passwords, file paths, or server details from being included in error messages, such information should only be stored in secured log files.

Using secure libraries : Rely on proven libraries and frameworks for error handling and logging that have already undergone security checks. This reduces the likelihood of implementation errors compromising security.

Conclusion

CWE-778 poses a severe security threat if bug reports are not adequately controlled. Developers must know the importance of handling errors securely to prevent unwanted information leaks. Applying secure programming practices, such as using design patterns to reuse error-handling logic and implementing centralised logging to detect attacks in real-time, can significantly increase the security and robustness of Java applications.

Secure error handling improves an application’s robustness and user experience by providing clear and useful instructions without overwhelming the user with technical details. The combination of security and usability is essential for the success and security of modern applications.

Ultimately, control over bug reports is integral to a software project’s overall security strategy. Bug reports can either be a valuable resource for developers or, if handled poorly, become a vulnerability for attackers to exploit. Disciplined error handling, modern design patterns, and attack detection technologies are critical to ensuring that error reports are used as a tool for improvement rather than a vulnerability.

]]>Design PatternJavaSecure Coding PracticesSecurityVaadinhttps://svenruppert.com/posts/code-security-through-unit-testing-the-role-of-secure-coding-practices-in-the-development-cycle/Wed, 16 Oct 2024 22:17:04 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/code-security-through-unit-testing-the-role-of-secure-coding-practices-in-the-development-cycle/Unit testing is an essential software development concept that improves code quality by ensuring that individual units or components of a software function correctly. Unit testing is crucial in Java, one of the most commonly used programming languages. This article will discuss what unit testing is, how it has evolved, and what tools and best practices have been established over the years.<![CDATA[

Unit testing is an essential software development concept that improves code quality by ensuring that individual units or components of a software function correctly. Unit testing is crucial in Java, one of the most commonly used programming languages. This article will discuss what unit testing is, how it has evolved, and what tools and best practices have been established over the years.

1. Definition von Unit Testing
2. The history of unit testing
   1. The 1990s and the emergence of JUnit
   2. The 2000s: Agile Methods and Test-Driven Development (TDD)
   3. The 2010s: Integration into CI/CD pipelines and the importance of automation
3. Tools and frameworks for unit testing in Java
4. Best practices for unit testing in Java
5. The future of unit testing
6. What are the disadvantages of unit testing?
   1. Time expenditure and costs
   2. Limited test coverage
   3. False sense of security
   4. Excessive mocking
   5. Difficulties in testability of legacy code
   6. Not suitable for complex test cases
   7. Test-driven development (TDD) can lead to excessive focus on details
   8. Test maintenance for rapid changes
   9. Lack of testing strategies in complex systems
   10. Additional effort without immediate benefit
   11. Conclusion
7. Does Secure Pay Load Testing belong to the area of ​​unit testing?
   1. What is Secure Payload Testing?
   2. Difference between Unit Testing and Secure Payload Testing
   3. Where does Secure Payload Testing fit in?
   4. Can Secure Payload Testing be part of Unit Testing?
8. Which secure coding practices play a role in connection with unit testing?
   1. Input validation and sanitisation
   2. Limit value analysis (boundary testing)
   3. Secure error handling
   4. Avoiding hard-coded secrets
   5. Use of safe libraries and dependencies
   6. Ensure encryption
   7. Least Privilege Principle
   8. Avoiding race conditions
   9. Avoidance of buffer overflows
   10. Safe use of third-party libraries

Definition von Unit Testing

Unit Testing refers to testing a program’s most minor functional units—usually individual functions or methods. A unit test ensures that a specific code function works as expected and typically checks whether a method or class returns the correct output for particular inputs.

In Java, unit testing often refers to testing individual methods in a class, testing whether the code responds as expected to both average and exceptional input.

Example in Java:

javaCopy

publicclassCalculator{    publicintadd(inta,intb){        returna+b;    }}

A unit test for theadd method could look like this:

javaCopy

publicclassCalculatorTest{    @Test    publicvoidtestAdd(){        Calculatorcalc=newCalculator();        assertEquals(5,calc.add(2,3));    }}

This example tests whether theadd method correctly combines two numbers. If the test is successful, the process works as expected.

The history of unit testing

The roots of unit testing lie in the early days of software development. As early as the 1960s, programmers recognised that it was essential to ensure that individual parts of a program were working correctly before integrating the entire system. However, unit testing only became primarily standardised in the 1990s with the spread of object-oriented programming languages ​​such as Java and C++.

The 1990s and the emergence of JUnit

One of the most influential events in the history of unit testing was the introduction of the frameworkJUnit for the Java programming language in 1999. Developed by Kent Beck and Erich Gamma, JUnit revolutionised how Java developers test their software.

JUnit enabled developers to write simple and easy-to-maintain unit tests for their Java programs. Before JUnit, no widely used, standardised tools facilitated unit testing in Java. Programmers had to create their test suites manually, which was tedious and error-prone.

JUnit allowed developers to automate tests, create test suites, and integrate testing into their development workflow. Its introduction contributed significantly to raising awareness about the importance of testing and promoting the adoption of unit testing in the industry.

The 2000s: Agile Methods and Test-Driven Development (TDD)

In the early 2000s, agile development methodologies gained popularity, particularlyExtreme Programming (XP) andScrum. These methods emphasized short development cycles, continuous integration, and, most importantly, testing. One of the core ideas of XP isTest-Driven Development (TDD) , a methodology in which tests are created before actual code is written.

In TDD, the development process is controlled by tests:

1. First, the developer writes a unit test that fails because the function to be tested still needs to be implemented.

2. The minimal code required to pass the test is then written.

3. Finally, the code is refined and optimised, with testing ensuring that existing functionality is not affected.

TDD requires developers to engage intensively with unit testing, which drives the entire development process. Unit tests are not just a way to ensure quality but an integral part of the design.

JUnit played a central role during this time, making implementing TDD in Java much more accessible. Developers could write and run tests quickly and easily, making the workflow more efficient.

The 2010s: Integration into CI/CD pipelines and the importance of automation

With the advent ofContinuous Integration (CI) andContinuous Delivery (CD) In the 2010s, unit testing became even more critical. CI/CD pipelines allow developers to continuously integrate changes to the code base into the central repository and automatically deploy new versions of their software.

In a CI/CD environment, automated testing, including unit testing, is critical to ensure that new code changes do not break existing functionality. Unit tests are typically the first stage of testing executed in a CI pipeline because they are fast and focused.

For this purpose, numerous tools have been created that integrate unit tests into the CI/CD workflow. Besides JUnit, various build tools, likeMaven andGradle , can run unit tests automatically. Test reports are generated, and the developer is notified if a test fails.

Another significant advancement during this time was integratingcode coverage tools likeJaCoCo. These tools measure the percentage of code covered by unit tests and help developers ensure that their tests cover all relevant code paths.

Tools and frameworks for unit testing in Java

JUnit : Probably the best-known and most widely used testing framework for Java. Since its introduction in 1999, JUnit has evolved and offers numerous features that make testing easier. With the introduction of JUnit 5, the framework became more modular and flexible.

TestNG : Another popular testing framework that offers similar functionality to JUnit but supports additional features such as dependency management and parallel test execution. TestNG is often used in larger projects that require complex testing scenarios.

Mockito : A mocking framework used to simulate dependencies of classes in unit tests. Mockito allows developers to “mock” objects to control the behaviour of dependencies without actually instantiating them.

JaCoCo : A tool for measuring code coverage. It shows what percentage of the code is covered by tests and helps developers identify untested areas in the code.

Gradle and Maven : These build tools provide native support for unit testing. They allow developers to run tests and generate reports during the build process automatically.

Best practices for unit testing in Java

Isolated testing : Each unit test should test a single method or function in isolation. This means that external dependencies such as databases, networks or file systems should be “mocked” to ensure that the test focuses only on the code under test.

Write simple tests : Unit tests should be simple and easy to understand. You should only test a single functionality and not have any complex logic or dependencies.

Regular testing : Tests should be conducted regularly, especially before every commit or build. This ensures that errors are detected early and that the code remains stable.

Test cases for limits : It is essential to test typical use cases and cover limit values ​​(e.g., minimum and maximum input values) and error cases.

Ensure test coverage : Code coverage tools like JaCoCo help developers ensure that most of the code is covered by tests. However, high test coverage should be one of many goals. The quality of the tests is just as important as the quantity.

The future of unit testing

Unit testing has constantly evolved over the past few decades, and this development is expected to continue in the future. The advent of technologies likeArtificial Intelligence (AI) andMachine Learning could open up new ways of generating and executing unit tests.

AI-powered tools could automatically generate unit tests by analysing the code and suggesting tests based on typical patterns. This could further automate the testing process and give developers more time for actual development.

Another trend is the increasing integration ofMutation Testing. This technique involves making minor changes to the code to check whether the existing tests detect the mutated code paths and throw errors. This allows developers to ensure that their tests cover code and detect errors.

Unit testing is an essential part of modern software development and has evolved from an optional practice to a standard. In Java, JUnit has contributed significantly to the proliferation and standardisation of unit testing. At the same time, agile methodologies such as TDD have integrated the role of testing into the development process.

With increasing automation through CI/CD pipelines and the availability of powerful tools, unit testing will continue to play a critical role in ensuring code quality in the future. Developers integrating unit testing into their workflow early on benefit from stable, maintainable and error-free code.

What are the disadvantages of unit testing?

Unit testing has many advantages, especially ensuring code quality and finding errors early. However, there are also some disadvantages and limitations that must be taken into account in practice:

Time expenditure and costs

Creation and maintenance : Writing unit tests requires additional time, especially in the early stages of a project. This effort increases development costs and can seem disproportionate for small projects.

Maintenance : When the code changes, the associated tests often also need to be adapted. This can be very time-consuming for larger code bases. Changes to the design or architecture can lead to numerous test changes.

Limited test coverage

Only tests isolated units : Unit tests test individual functions or methods in isolation. They do not cover the interaction of different modules or layers of the system, which means that they cannot reveal integration problems.

Not suitable for all types of errors : Unit tests are good at finding logical errors in individual methods, but they cannot detect other types of errors, such as errors in interaction with databases, networks, or the user interface.

False sense of security

High test coverage ≠ freedom from errors : Developers may develop a false sense of security when achieving high test coverage. Just because many tests cover the code doesn’t mean it’s bug-free. Unit tests only cover the specific code they were written and may not test all edge or exception cases.

Blind trust in testing : Sometimes, developers rely too heavily on unit testing and neglect other types of testing, such as integration testing, system testing, or manual testing.

Excessive mocking

Mocks can distort reality : When testing classes or methods that depend on external dependencies (e.g. databases, APIs, file systems), mock objects are often used to simulate these dependencies. However, extensive mocking frameworks such as Mockito can lead to unrealistic tests that work differently than the system in a real environment.

Complex dependencies : When a class has many dependencies, creating mocks can become very complicated, making the tests difficult to understand and maintain.

Difficulties in testability of legacy code

Legacy-Code ohne Tests : Writing unit tests can be challenging and time-consuming in existing projects with older code (legacy code). Such systems may have been designed without testability in mind, making unit tests difficult to write.

Refactoring necessary : Legacy code often needs to be refactored to enable unit testing, which can introduce additional risks and costs.

Not suitable for complex test cases

Not suitable for end-to-end testing : Unit tests are designed to test individual units and are not intended to cover end-to-end test cases or user interactions. Such testing requires other types of testing, such as integration, system, or acceptance testing.

Limited perspective : Unit tests often only consider individual system components and not the behaviour of the entire system in real usage scenarios.

Test-driven development (TDD) can lead to excessive focus on details

Design of code influenced by testing : In TDD, the emphasis is on writing tests before writing the code. This can sometimes lead to developers designing code to pass tests rather than developing a more general, robust solution.

Excessive focus on detailed testing : TDD can cause developers to focus too much on small details and isolated components instead of considering the overall system architecture and user needs.

Test maintenance for rapid changes

Frequent changes lead to outdated tests : In fast-paced development projects where requirements and code change frequently, tests can quickly become obsolete and unnecessary. Maintaining these tests can become significant without providing any clear added value.

Tests as ballast : If code is constantly evolving and tests are not updated, outdated or irrelevant tests can burden the development process.

Lack of testing strategies in complex systems

Complexity of test structure : It can be difficult to develop a meaningful unit testing strategy that covers all aspects of very complex systems. This often results in fragmented and incomplete tests or inadequate testing of critical areas of the system.

Testing complexity in object-oriented designs : For highly object-oriented programs, it can be difficult to identify precise units for unit testing, especially if the classes are highly interconnected. In such cases, writing unit tests can become cumbersome and inefficient.

Additional effort without immediate benefit

Cost-benefit analysis for small projects : In small projects or prototypes where the code is short-lived, the effort spent writing unit tests may outweigh the benefits. In such cases, other testing methods, such as manual or simple end-to-end testing, may be more efficient.

Conclusion

Although unit testing has numerous advantages, there are also clear disadvantages that must be considered. The additional time required, limited test coverage, and potential maintenance challenges must be weighed when planning a testing process. Unit testing is not a panacea but should be viewed as one tool among many in software development. However, combined with other types of testing and a well-thought-out testing strategy, unit testing can significantly improve code quality.

Does Secure Pay Load Testing belong to the area of ​​unit testing?

Secure Payload Testing usually belongs to a different area than the traditional area ofUnit Testing, which isSecurity testing and partialIntegration Tests. Let’s take a closer look at this to better understand the boundaries.

What is Secure Payload Testing?

Secure Payload Testing refers to testing security-related data or messages exchanged between system components. This particularly applies to scenarios where sensitive data (such as passwords, API keys, encrypted data, etc.) must be protected and handled correctly in communication between systems. It tests whether data is appropriately encrypted, decrypted and authenticated during transmission and whether there are any potential security gaps in handling this data.

Examples of typical questions in secure payload testing are:

• Is sensitive data encrypted and decrypted correctly?
• Does data remain secure during transmission?
• Is it ensured that the payload data does not contain security holes, such as SQL injections or cross-site scripting (XSS)?
• Is the integrity of the payload guaranteed to prevent manipulation?

Difference between Unit Testing and Secure Payload Testing

Unit Testing focuses on theFunctionality of individual program components or methods in isolation**.** It usually checks whether a method delivers the expected outputs for certain inputs. The focus is on the correctness and stability of the program’s logical units, not directly on the security or protection of data.

An example of a unit test in Java would be testing a simple mathematical function. The unit test would check whether the method works correctly. Security aspects, such as handling confidential data or ensuring encryption, are usually outside of such tests.

In contrast,Secure Payload Testing involves the secure handling and processing of data during transmission or storage. This is often part ofsecurity testing , which aims to ensure that data is properly protected and not vulnerable to attacks or data leaks.

Where does Secure Payload Testing fit in?

Integration tests : Secure Payload Testing could be part of integration testing, which tests the interaction between different components of a system, e.g., between a client and a server. Here, one would ensure that the payload is properly encrypted and the transmission over the network is secure.

Security testing : In more complex systems, secure payload testing belongs more toSecurity testing , which involves attacks on data security, integrity, and confidentiality of payloads. These tests often go beyond the functionality of individual code units and require special testing strategies, such as penetration testing or testing for known security vulnerabilities.

End-to-End Tests : Since Secure Payload Testing is often related to data transfer, it can also be part ofEnd-to-End tests. The entire system is tested here, from input to processing to output. These tests check whether the data is encrypted correctly at the beginning and decrypted and processed correctly at the end.

Can Secure Payload Testing be part of Unit Testing?

In special cases,an aspect of secure payload testing can be part of unit tests, especially if the security logic is very closely linked to the functionality of the unit under test (e.g., an encryption or decryption method).

An example could be testing anEncryption method in isolation:

javaCopy

publicStringencrypt(Stringdata,Stringkey){    // Encryption logic    returnencryptedData;}publicStringdecrypt(StringencryptedData,Stringkey){    // Decryption logic    returndecryptedData;}

Here, you could write unit tests that check whether:

• The text is correctly encrypted and decrypted.
• For the same input data, the same output is always produced (in the case of deterministic encryption).
• The method responds correctly to invalid inputs (e.g. wrong key, invalid data format).

Despite these specific test cases, secure payload testing generally focuses on not only isolated functionality but also security and integrity in the context of other system components.

Secure Payload Testing does not belong to classicUnit Testing. It typically concerns security, integrity, and correct data processing in a broader context, often involving the interaction of multiple system components. It falls more into the realm ofsecurity andintegration tests.

However, unit tests can test parts of security logic to some extent, especially when encryption or security functions are to be tested in isolation. However, the overall picture of security requirements and protection of payloads is usually determined by more comprehensive testing, such asIntegration Tests ,Security testing, orEnd-to-End tests.

Which secure coding practices play a role in connection with unit testing?

Secure Coding Practices are essential to ensure the code is secure against potential attacks and vulnerabilities. These practices are fundamental to ensuring software security, and they can be closely related toUnit Testing. While unit testing primarily aims to verify code’s functionality, secure coding practices can help ensure that code is robust and secure. Here are some of the key Secure Coding Practices related to Unit Testing:

Input validation and sanitisation

Safe practice : Always ensure that input from external sources (user input, API calls, file input) is validated and “sanitised” to avoid dangerous content such as SQL injection or cross-site scripting (XSS).

Connection to unit testing : Unit tests should ensure that methods and functions respond correctly to invalid or potentially dangerous input. Tests should contain inputs such as unexpected special characters, entries that are too long or short, empty fields, or formatting errors.

Example unit test in Java :

xmlCopy

 @Test     public void testValidateUserInput() {         String invalidInput = "<script>alert('xss')</script>";         assertFalse(InputValidator.isValid(invalidInput));     }

This test checks whether theisValid method correctly rejects unsafe input as invalid.

Limit value analysis (boundary testing)

Safe practice : Inputs should be tested against their maximum and minimum limits to ensure the code does not crash or be vulnerable to buffer overflows.

Connection to unit testing : Unit tests should ensure the application safely responds to input at the top and bottom of its allowed ranges. This helps prevent typical attacks such as buffer overflows.

Example : If a function only accepts a certain number of characters, the unit test should check how the function responds to inputs that are precisely at or above that limit.

Secure error handling

Safe practice : Error handling should not reveal sensitive information, such as stack traces or details about the application’s internal structure, as attackers can exploit such information.

Connection to unit testing : Unit tests should ensure errors and exceptions are handled correctly without exposing sensitive information. Unit tests can trigger targeted exception situations and check whether only safe and user-friendly error messages are returned.

Example :

javaCopy

@Test     publicvoidtestHandleInvalidInputGracefully(){         try{             myService.processUserInput(null);         }catch(Exceptione){             fail("Exception thrown, but should have been handled gracefully.");         }     }

Avoiding hard-coded secrets

Safe practice : Never hard-code sensitive information such as passwords, API keys, or tokens in code. Instead, such data should be stored in secure environment variables or configuration files.

Connection to unit testing : Unit tests should ensure that sensitive data is handled securely. They should also check that the code loads external configuration sources correctly and does not accidentally use hard-coded secrets.

Example :

javaCopy

@Test     publicvoidtestExternalConfigForSecrets(){         assertNotNull(System.getenv("API_KEY"));     }

Use of safe libraries and dependencies

Safe practice : Users should take care to use secure libraries and frameworks and update them regularly to avoid known security vulnerabilities.

Connection to unit testing : Unit tests should ensure the libraries are correctly integrated and updated. Testing functionality that depends on external libraries is also essential to ensure that security mechanisms in those libraries are used correctly.

Ensure encryption

Safe practice : Sensitive data should be stored and transmitted in encrypted form to prevent unauthorised access or data leaks.

Connection to unit testing : Unit tests should verify that data is encrypted and decrypted correctly. For example, a test could ensure that the encryption and decryption methods work consistently and without errors.

Example :

javaCopy

@Test     publicvoidtestEncryptionAndDecryption(){         Stringoriginal="Sensitive Data";         Stringencrypted=encrypt(original,"mySecretKey");         Stringdecrypted=decrypt(encrypted,"mySecretKey");         assertEquals(original,decrypted);     }

Least Privilege Principle

Safe practice : Methods and functions should only be executed with the minimum rights and access required.

Connection to unit testing : Unit tests should ensure that methods only work with the minimum required data and that no unauthorised access to resources is possible. For example, tests could check whether protected resources are only accessed after successful authentication.

Example :

javaCopy

@Test     publicvoidtestUnauthorizedAccess(){         assertThrows(AccessDeniedException.class,()->{             userService.deleteUserDataWithoutPermission();         });     }

Avoiding race conditions

Safe practice : Race conditions can occur when multiple threads or processes access shared resources simultaneously. They should be avoided to prevent security issues such as unpredictable behavior or data corruption.

Connection to unit testing : Unit tests should ensure that the code is thread-safe and that no race conditions occur. This can be verified by testing code under multiple access or by using mock threads.

Example :

javaCopy

@Test     publicvoidtestConcurrentAccess()throwsInterruptedException{         CountDownLatchlatch=newCountDownLatch(2);         Runnabletask=()->{             sharedResource.modify();             latch.countDown();         };         Threadt1=newThread(task);         Threadt2=newThread(task);         t1.start();         t2.start();         latch.await();         assertTrue(sharedResource.isConsistent());     }

Avoidance of buffer overflows

Safe practice : Buffer overflows occur when a program writes more data into a memory area than it can hold. Although Java is less prone to buffer overflows than C or C++, thanks to automatic memory management, care should still be taken to ensure that arrays and memory structures are used safely.

Connection to unit testing : Unit tests should test edge cases and maximum input values ​​to ensure that overflows do not occur.

Safe use of third-party libraries

Safe practice : Third-party libraries should be used safely and regularly checked for known vulnerabilities.

Connection to unit testing : Tests can ensure that functions and classes from third-party libraries are implemented correctly and used safely. Mocking can be used to simulate external dependencies safely.

Secure Coding Practices play an essential role inUnit Testing , as they ensure that the code is not only functionally correct but also secure against potential attacks. Unit tests should aim to check security-relevant aspects such as input validation, secure error handling, encryption, and rights assignment. By incorporating these practices into the unit testing process, developers can ensure their applications are robust, secure, and protected against many common attack patterns.

]]>JavaSecure Coding PracticesTDDhttps://svenruppert.com/posts/understanding-toctou-time-of-check-to-time-of-use-in-the-context-of-cwe-377/Mon, 07 Oct 2024 17:57:36 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/understanding-toctou-time-of-check-to-time-of-use-in-the-context-of-cwe-377/Building on the discussion of “CWE-377: Insecure Temporary File”, it’s essential to delve deeper into one of the most insidious vulnerabilities that can arise in this context—TOCTOU (Time-of-Check to Time-of-Use) race conditions. TOCTOU vulnerabilities occur when there is a time gap between verifying a resource (such as a file) and its subsequent use. Malicious actors can exploit this gap, especially in temporary file scenarios, leading to serious security breaches. This follow-up article will explore how TOCTOU conditions manifest in software, particularly in managing temporary files, and discuss strategies to mitigate these risks to ensure robust and secure application development.<![CDATA[

Building on the discussion of “CWE-377: Insecure Temporary File”, it’s essential to delve deeper into one of the most insidious vulnerabilities that can arise in this context—TOCTOU (Time-of-Check to Time-of-Use) race conditions. TOCTOU vulnerabilities occur when there is a time gap between verifying a resource (such as a file) and its subsequent use. Malicious actors can exploit this gap, especially in temporary file scenarios, leading to serious security breaches. This follow-up article will explore how TOCTOU conditions manifest in software, particularly in managing temporary files, and discuss strategies to mitigate these risks to ensure robust and secure application development.

I wrote an article about CWE-377 itself. You can find it here:https://svenruppert.com/2024/08/21/cwe-377-insecure-temporary-file-in-java/

TOCTOU (Time-of-Check to Time-of-Use) is a type of race condition that occurs when the state of a resource (such as a file, memory, or variable) is checked (validated or verified) and then used (modified or accessed) in separate steps. If an attacker can alter the resource between these two steps, they may exploit the gap to introduce malicious behaviour or compromise the security of an application.

How TOCTOU Applies to Temporary Files

In the context of temporary file creation, TOCTOU vulnerabilities arise when the program checks whether a temporary file exists and then creates or opens it. If an attacker manages to create a file with the same name in the interval between these operations, they can control the contents or properties of the file the program thinks it is safely creating or accessing.

For example, consider the following sequence of operations:

Check if a file with a specific name exists : The program checks if a temporary file (e.g.,tempfile.txt) already exists.

Create or open the file : If the file does not exist, the program creates or opens it.

If an attacker creates a file namedtempfile.txt in the time between the check and the creation, the program may inadvertently interact with the attacker’s file instead of a legitimate, secure file. This can lead to issues such as unauthorised data access, corruption, or privilege escalation.

Detailed Example of TOCTOU Vulnerability

Consider the following Java code:

javaCopy

importjava.io.File;importjava.io.IOException;publicclassTOCTOUVulnerabilityExample{    publicstaticvoidmain(String[]args)throwsIOException{        FiletempFile=newFile("/tmp/tempfile.txt");        // Time-of-check: Verify if the file exists        if(!tempFile.exists()){            // Time-of-use: Create the file            tempFile.createNewFile();            System.out.println("Temporary file created at: "+tempFile.getAbsolutePath());        }    }}

In this example:

1. The program first checks if**tempfile.txt** exists using the**exists()** method.

2. If the file does not exist, it creates a new file with the same name using the**createNewFile()** method.

The vulnerability here lies between the time the**exists()** check is performed and the time the**createNewFile()** method is called. If an attacker creates a file named**tempfile.txt** between these two steps, the program will not create a new file but instead interact with the attacker’s file, potentially leading to a security breach.

Exploitation of TOCTOU in Temporary Files

An attacker can exploit TOCTOU in several ways:

File Pre-Creation : The attacker creates a file with the same name as the intended temporary file in a directory accessible by the application. If the file permissions are weak, the attacker may gain control over the contents of this file.

Symlink Attack : The attacker can create a symbolic link (symlink) that points to a sensitive file (e.g.,/etc/passwd) with the same name as the temporary file. When the program tries to write to or read from the temporary file, it might access the sensitive file instead, leading to data corruption or information leakage.

Privilege Escalation : If the program runs with elevated privileges (e.g., as root), an attacker could exploit the TOCTOU race condition to modify files or data that they otherwise would not have permission to access or change.

Preventing TOCTOU Vulnerabilities in Java

To prevent TOCTOU vulnerabilities, particularly when dealing with temporary files, developers should follow best practices that minimise the risk of a race condition:

Use Atomic Operations

Atomic operations are inseparable; they either complete entirely or do not happen at all, leaving no opportunity for an attacker to intervene. Java’sFile.createTempFile() method is atomic when creating temporary files. This means that the file creation and name generation occur in a single step, eliminating the TOCTOU window.

javaCopy

importjava.io.File;importjava.io.IOException;publicclassAtomicTempFileCreation{    publicstaticvoidmain(String[]args)throwsIOException{        // Atomic operation to create a temporary file        FiletempFile=File.createTempFile("tempfile_",".tmp");        tempFile.deleteOnExit();        System.out.println("Secure temporary file created at: "+tempFile.getAbsolutePath());    }}

Here,**File.createTempFile()** ensures that the file is both uniquely named and securely created without exposing the application to race conditions.

Use Secure Directories

Place temporary files in a secure, private directory that is inaccessible to other users. This limits attackers’ opportunities to exploit TOCTOU vulnerabilities because they cannot easily place or manipulate files in these directories.

LeverageFiles andPath (NIO.2 API)

Java’s NIO.2 API (java.nio.file) offers more advanced file-handling mechanisms, including atomic file operations. For instance,**Files.createTempFile()** allows for atomic file creation with customisable file attributes, such as secure permissions, further reducing the risk of TOCTOU vulnerabilities.

javaCopy

importjava.nio.file.Files;importjava.nio.file.Path;importjava.nio.file.attribute.PosixFilePermissions;importjava.util.Set;publicclassSecureAtomicTempFile{    publicstaticvoidmain(String[]args)throwsIOException{        // Create a temporary file with atomic operations and secure permissions        PathtempFile=Files.createTempFile("secure_tempfile_",".tmp",             PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rw-------")));        System.out.println("Secure temporary file created at: "+tempFile.toAbsolutePath());    }}

This approach combines atomic file creation with restrictive file permissions, mitigating both TOCTOU vulnerabilities and other potential security risks.

Conclusion

TOCTOU vulnerabilities represent a significant security risk when handling temporary files, particularly if these files are created in an insecure or non-atomic manner. The key to preventing these vulnerabilities lies in eliminating the gap between the time-of-check and time-of-use, typically by using atomic file creation methods provided by secure APIs, such as**File.createTempFile()** or**Files.createTempFile()**.

By understanding the risks associated with TOCTOU race conditions and following best practices, developers can ensure that their Java applications are resilient to these attacks and maintain the software’s integrity and security.

]]>JavaSecure Coding PracticesSecurityhttps://svenruppert.com/posts/bld-a-lightweight-java-build-tool/Thu, 26 Sep 2024 17:31:17 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/bld-a-lightweight-java-build-tool/What is a dependency management tool? A dependency management tool is a software system or utility that automates the process of identifying, retrieving, updating, and maintaining the external libraries or packages (referred to as dependencies) required by a software project. It ensures that all necessary dependencies are included and managed in a standardised way, which helps prevent version conflicts, missing libraries, and manual errors during software development.<![CDATA[

What is a dependency management tool?

Adependency management tool is a software system or utility that automates the process of identifying, retrieving, updating, and maintaining the external libraries or packages (referred to asdependencies) required by a software project. It ensures that all necessary dependencies are included and managed in a standardised way, which helps prevent version conflicts, missing libraries, and manual errors during software development.

1. What is a dependency management tool?
   1. Dependency Identification:
   2. Fetching Dependencies:
   3. Version Management:
   4. Scope & Environment Management:
   5. Build Process Integration:
   6. Popular Dependency Management Tools:
   7. Benefits of Dependency Management Tools:
   8. Example Workflow:
2. The classical Dependency manager for Java - Maven
   1. Project Structure andpom.xml
   2. Build Lifecycle
   3. Dependency Management
   4. Maven Repositories
   5. Plugins
   6. How It All Works Together:
3. BLD - the lightweight Java Build System
   1. Key Features of bld
   2. How bld Stands Out in the Java Ecosystem
   3. Usage Example
4. The difference between Maven and BLD
   1. Build Language:
   2. Declarative vs. Code-Based:
   3. Task Execution:
   4. IDE Integration:
   5. Dependency Management:
   6. Build Artifacts:
   7. Complexity and Learning Curve:
   8. Performance:
5. How to start with bld?
6. An Example migration of a project from Maven to bld
   1. The Maven Project Structure
7. Conclusion

Demo - Project are available here:https://github.com/Java-Publications/Blog---Core-Java---2024.08---bld-the-lightweight-build-tool

Here’s how dependency management tools generally function:

Dependency Identification:

Developers list the libraries or frameworks their project depends on in a configuration file (like Maven’s pom.xml or Gradle’s build.gradle). Each dependency is defined by attributes such asgroup ID ,artifact ID , andversion.

Example (Maven) :

xmlCopy

<dependency>       <groupId>junit</groupId>       <artifactId>junit</artifactId>       <version>4.12</version>   </dependency>

Fetching Dependencies:

The tool retrieves the required dependencies fromrepositories (such as Maven Central or custom/private repositories) and downloads them to alocal repository on the developer’s machine. These repositories host many external libraries, frameworks, and plugins.

Local Repository : A cache stored locally on a developer’s machine to prevent repeated downloads.

Remote Repository : Centralized or custom servers hosted by the dependencies (e.g., Maven Central).

Version Management:

Dependency management tools handledependency versioning , ensuring the correct versions of external libraries are used. Tools like Maven or Gradle supporttransitive dependency resolution , meaning if a dependency relies on another library, that secondary library will also be fetched automatically.

Version Conflicts : If two libraries depend on different versions of the same dependency, the tool resolves the conflict by selecting the nearest version in the dependency tree or the explicitly defined one.

Scope & Environment Management:

Dependency management tools allow for differentscopes orprofiles that specify when a dependency is required (e.g., only during testing or in a production environment). This helps optimise resource usage and prevent unnecessary dependencies from being included in the final build.

Example Scopes :

Compile : Dependencies available during compilation and runtime.

Test : Dependencies used only during testing (e.g., JUnit).

Provided : Dependencies expected to be supplied by the runtime environment.

Build Process Integration:

Dependency management tools integrate into a project’s build lifecycle. They ensure that the right versions of libraries are fetched before compiling, packaging, or testing the application. For instance, Maven’s**install** phase ensures that all dependencies are installed in the local repository before building.

Popular Dependency Management Tools:

Maven : Uses**pom.xml** to define dependencies and follows a declarative model with predefined lifecycles.

Gradle : Uses Groovy or Kotlin DSL for its build scripts, offering more flexibility and allowing for custom build logic.

npm (Node Package Manager) : Manages dependencies for JavaScript and Node.js projects, using a**package.json** file.

bld : A newer tool for Java projects that allows developers to define dependencies directly in Java code.

Benefits of Dependency Management Tools:

Automation : They automate the retrieval and management of libraries, removing the need for developers to download and include them manually.

Consistency : Ensures that every developer in a team uses identical versions of dependencies, leading to more consistent builds.

Conflict Resolution : Handles version conflicts between dependencies automatically, preventing runtime errors.

Efficiency : By caching dependencies locally, tools reduce repeated downloads and improve build times.

Security : Modern tools often check for security vulnerabilities in dependencies and alert developers to updates.

Example Workflow:

A developer declares dependencies in a configuration file.

The tool checks the local repository for cached versions of the libraries.

If dependencies aren’t cached, the tool fetches them from a remote repository.

The tool ensures all dependencies are included during the build process (compile, test, package, etc.).

Dependency management tools are vital for modern software development, especially in large projects with numerous external libraries. They automate the complex process of handling dependencies, reducing manual errors, improving efficiency, and ensuring the project builds correctly across different environments.

The classical Dependency manager for Java - Maven

Maven is a build automation and project management tool primarily used in Java projects. It uses adeclarative model to manage a project’s build, reporting, and dependencies using an XML configuration file called**pom.xml**. Below is an overview of how Maven works:

Project Structure andpom.xml

At the heart of every Maven project is the**pom.xml** file (Project Object Model). This file defines the project structure, dependencies, build plugins, and goals. Here’s what**pom.xml** generally contains:

Group ID : Identifies the project’s group (often based on package naming).

Artifact ID : A unique identifier for the project.

Version : Version of the project being built.

Dependencies : A list of libraries or frameworks the project requires (such as**JUnit** for testing or**Spring** for dependency injection).

Plugins : Additional tools that enhance the build process (for tasks like creating JAR files, running tests, etc.).

xmlCopy

<project>       <modelVersion>4.0.0</modelVersion>       <groupId>com.example</groupId>       <artifactId>my-app</artifactId>       <version>1.0-SNAPSHOT</version>       <dependencies>           <dependency>               <groupId>junit</groupId>               <artifactId>junit</artifactId>               <version>4.12</version>               <scope>test</scope>           </dependency>       </dependencies>   </project>

Build Lifecycle

Maven operates on a series oflifecycles. Thedefault lifecycle consists of a sequence of build phases that are invoked when building a project:

• validate : Check if the project is correct and all necessary information is available.
• compile : Compile the source code of the project.
• test : Run tests on the compiled code using a suitable testing framework (e.g., JUnit).
• package : Package the compiled code into a distributable format, like a JAR or WAR.
• install : Install the package into the local repository for use as a dependency in other local projects.
• deploy : Copy the final package to the remote repository for sharing with other developers.

These lifecycle phases are sequential, meaning invoking**install** will also execute all phases before it (from**validate** to**package**).

Dependency Management

Maven’s most powerful feature is itsdependency management system. Maven fetches external libraries fromMaven Central , a global repository that hosts many open-source libraries. Maven usestransitive dependency resolution , meaning that if Project A depends on Library B and Library B depends on Library C, Maven will automatically fetch Library C.

Dependencies are defined in the**pom.xml** under the**< dependencies>** section, as shown above. Maven manages and can automatically update versions, making integrating libraries very efficient.

Maven Repositories

Maven uses three types of repositories:

Local Repository : This is on the developer’s machine. When Maven builds a project, it checks the local repository first to resolve dependencies.

Central Repository : Maven Central is the default remote repository where Maven looks for dependencies that are not available in the local repository.

Remote Repository : Additional remote repositories, such as private or custom repositories that host specific artefacts, can be defined.

Plugins

Plugins in Maven extend its functionality. Some common Maven plugins include:

maven-compiler-plugin : Compiles the source code.

maven-surefire-plugin : Runs unit tests.

maven-jar-plugin : Packages the project as a JAR file.

maven-war-plugin : Packages the project as a WAR file for web applications.

These plugins are usually added to the**pom.xml** under the**< build>** section.

Profiles

Maven also supports profiles, which allow developers to define different configurations for different environments (e.g., development, testing, production). Profiles are specified in thepom.xm l and can be activated by different triggers, such as system properties or environments.

xmlCopy

<profiles>       <profile>           <id>dev</id>           <properties>               <env>development</env>           </properties>       </profile>   </profiles>

Multi-Module Projects

Maven can handlemulti-module projects , where a single project is divided into multiple sub-projects or modules. Each module typically has its own**pom.xml** but is coordinated by a parent POM. This feature is helpful for large projects with independent yet related components.

How It All Works Together:

1. The developer defines the project structure and dependencies in thepom.xml.

2. Maven processes this file and resolves all dependencies by downloading required JAR files from remote repositories (if they are not found in the local repository).

3. Maven executes phases from the build lifecycle, compiling, testing, and packaging the project.

4. Plugins add extra functionality, such as running tests, generating documentation, and creating artefacts.

Maven’s declarative model and lifecycle management make it a popular choice for managing complex Java projects. Its rich plugin ecosystem and dependency management reduce manual effort while standardising the build process. However, the XML-heavy configuration can be cumbersome for some developers, especially compared to newer build tools likebld.

BLD - the lightweight Java Build System

bld is a lightweight build tool designed specifically for the Java ecosystem. Unlike traditional build tools like Maven or Gradle, which rely on declarative syntax and often involve complex configurations,bld allows developers to write their build logic in pure Java. This approach simplifies the learning curve and reduces the cognitive load on developers by enabling them to stay within the Java language for both application and build logic, making it a particularly modern and streamlined tool for Java projects.

Key Features of bld

Java-Based Build Logic : The most distinctive aspect ofbld is that it uses Java itself to define the build process. This eliminates the need for learning domain-specific languages (DSLs) or XML configurations as is the case with Maven or Gradle. Withbld , build logic becomes part of the Java code, making it easy to reason about and maintain in the same environment where application development occurs.

Explicit Task Execution :bld emphasises a transparent and predictable execution model. Tasks inbld do not run automatically but must be explicitly triggered, giving developers complete control over the build process. This removes any “auto-magical” behaviour often associated with other tools, which can sometimes lead to unexpected outcomes.

Simple Dependency Management : Dependency management inbld is intuitive and leverages Java constructs. Dependencies are managed through a standard library system, and they can be added directly in the Java build script by specifying the Maven coordinates. This system allowsbld to handle fetching and resolving dependencies automatically, similar to Maven or Gradle, but with less complexity.

Integration with Java 17 :bld takes full advantage of the latest Java features, relying on Java 17 as its minimum requirement. This allows developers to use modern Java syntax, including records, sealed classes, and pattern matching, to write concise and clean build scripts. The tool supports features like code auto-completion and Javadoc integration, making it easy to navigate within popular Integrated Development Environments (IDEs).

IDE Support :bld has integrations for popular IDEs like IntelliJ IDEA, offering features such as automatic project detection, quick access to the main build class, and the ability to run multiple build commands directly from the IDE. This smooths the workflow for developers who prefer using graphical interfaces over command-line interactions.

Command-Line Interface (CLI) : Thebld CLI offers a set of standard commands that allow developers to compile, run, test, and package their projects. It supports everyday build tasks like creating JAR files, managing dependencies, running tests, and generating Javadoc. Developers can also define custom commands within their build scripts using Java, extending the tool’s flexibility.

How bld Stands Out in the Java Ecosystem

In the Java build ecosystem, tools typically fall into two categories: declarative (like Maven) and code-based (like Gradle).bld is part of the second group but adds the advantage of being immediately executable without predefining build plans. This immediate execution reduces the complexity of managing and understanding the build pipeline.

For developers familiar with scripting environments like Node.js or Go,bld ’s ergonomics offer a familiar, streamlined experience. It delivers the ease of dependency management while maintaining transparency and direct control, helping to avoid common issues associated with over-engineered build systems.

Usage Example

A basicbld file for a Java project could look like this:

javaCopy

packagecom.example;importrife.bld.Project;importjava.util.List;import staticrife.bld.dependencies.Repository.*;import staticrife.bld.dependencies.Scope.*;publicclassMyappBuildextendsProject{    publicMyappBuild(){        pkg="com.example";        name="Myapp";        mainClass="com.example.MyappMain";        version=version(0,1,0);        repositories=List.of(MAVEN_CENTRAL);        scope(compile).include(dependency("com.fasterxml.jackson.core","jackson-databind",version(2,16,0)));    }}

In this setup, a developer specifies dependencies in the build file itself, alongside build configurations, removing the need for separate files like**pom.xml** in Maven.

bld offers a powerful yet simple alternative for Java developers who seek to avoid the complexities of traditional build tools. Writing build logic in Java, using an explicit task execution model, and supporting modern Java features provide a clear and concise way to manage builds, dependencies, and distribution. It suits developers who prefer a code-centric approach to their entire development lifecycle, from coding to building.

The difference between Maven and BLD

bld andMaven are both Java build tools, but they differ significantly in their approach, complexity, and usage. Here’s a comparison of the two:

Build Language:

Maven : Uses an XML-based configuration file called**pom.xml** (Project Object Model). This declarative approach requires developers to define a static configuration of dependencies, plugins, and tasks in a verbose XML format.

bld : It usesJava as the build scripting language, meaning developers write their build logic using Java code. This approach leverages Java’s advantages, such as type safety, auto-completion, and the ability to refactor using IDE tools.

Declarative vs. Code-Based:

Maven : Fully declarative. Developers declare what they want to happen, and Maven uses its lifecycle to determine how to execute tasks. This provides predictability but can be hard to customise beyond its standard behaviour.

bld : Primarilycode-based. Developers write Java code that is executed immediately. This provides more flexibility and control over the build process, allowing custom build flows and logic to be easily implemented.

Task Execution:

Maven : Its lifecycle uses predefined phases, such as**validate**,**compile**,**test**,**package**, and**install**. Each phase automatically triggers a set of goals, which is convenient for standard projects but can become cumbersome if you need custom behaviour outside of Maven’s predefined lifecycle.

bld : Offersexplicit control over tasks. Tasks are not automatically linked in a lifecycle; developers must call them explicitly, which provides more control and avoids the hidden behaviour common in Maven.

IDE Integration:

Maven : Has widespread support across IDEs like Eclipse, IntelliJ IDEA, and NetBeans. It has been around for a long time, so IDEs can automatically recognise Maven projects and provide support like dependency management, project configuration, and running lifecycle phases directly from the IDE.

bld : Offers integration withIntelliJ IDEA and leverages the build logic as Java code, providing features such as code navigation, auto-completion, and even IDE-run configurations.

Dependency Management:

Maven : Dependency management is one of Maven’s most vital features. It uses a central repository (Maven Central) and follows a transitive dependency model, automatically downloading dependencies and their sub-dependencies. However, this sometimes results in dependency conflicts or versioning issues.

bld : Similar to Maven,bld supports dependency management using Maven Central or other repositories. However, it allows developers to handle dependency logic within the Java build file itself, which can simplify certain aspects of dependency resolution.

Build Artifacts:

Maven : Offers a standardised way to build JAR, WAR, and other packages using plugins. It also supports additional artefact types like sources, documentation, and testing.

bld : Supports similar build artifacts (JAR, UberJAR, etc.) but gives developers more direct control over how these are generated. For example, developers can include custom commands to create specific outputs like precompiled templates.

Complexity and Learning Curve:

Maven : Can be complex, especially when dealing with multi-module projects or needing custom behaviour that requires a deep understanding of Maven’s plugin architecture and lifecycle phases. Modifying or extending Maven’s behaviour often involves writing custom plugins or extensive XML configuration.

bld : Aims to reduce complexity by allowing developers to manage builds using familiar Java code. It is simpler for Java developers to grasp, and writing custom build logic feels more natural compared to learning Maven’s XML configuration and plugins.

Performance:

Maven : Due to its lifecycle model and plugin architecture, Maven can be slower, mainly when performing complete builds with many plugins or complex dependency graphs. Incremental builds and caching are not as optimised as some newer tools.

bld : Providesfaster builds in many cases due to its simpler execution model and the immediate execution of Java code without going through multiple lifecycle phases.

How to start with bld?

The easiest way to start with bld is on commandline. For this go into the directory where you want to start with the project. Open a terminal and execute the following command that is available at the home page of the project:

sqlCopy

bash-c"$(curl -fsSL https://rife2.com/bld/create.sh)"Downloadingbldv2.1.0...Welcometobldv2.1.0.Pleaseenteranumberfortheprojecttype:  1:base  (Javabaselineproject)  2:app   (Javaapplicationproject)  3:lib   (Javalibraryproject)  4:rife2 (RIFE2webapplication)

This will download the bld-Script and execute it. Now, you can choose what kind of project you want to start. I am selecting1 for a baseline project. The next questions are package name and application name. With this informationes the generating process will be done. You will find a directory with the appname inside the directory. In my example it is the namecore.

Inside the app directory there is a script for unix/osx and one for windows. This is the wrapper for bld. I am using the osx version in my example here.

To test if the installation is correct, use the command./bld version. In my case the answer is 2.6.0.

There are two directories. The first one is calledlib. Inside there are different directories, one is calledbld and the others are named by the different scopes you know from the maven lifecycles. (compile ,provided ,runtime andtest). Later you will find the dependencies that are required for the different scopes inside these directories. The most important directory right now, is the directory calledbld. Inside the directory there is the build related files like the wrapper, properties and cache. For customizing the build environment, the properties file is the right place to go. Here you can activate extensions for example. But this will be dome later.

Back at the root directory of the project, there is thesrc folder as well. Inside the source folder are three subfolders. The src and test folder is known from maven already. The third folder calledbld contains the sources for the build configuration. The Class is called appname plus “Build” in this case it isCoreBuild.

Here we can start defining the build process itself.

An Example migration of a project from Maven to bld

Migrating a Maven project tobld involves several steps, asbld operates on Java-based build scripts instead of Maven’s**pom.xml** XML-based configuration. Here’s a general example of how you might approach migrating a Maven project tobld.

The Maven Project Structure

Before migrating, you need to analyse the Maven project structure:

**pom.xml**: This file contains dependencies, plugins, repositories, build profiles, and other configurations.

We will go through different sections of the pom.xml and transforming them into the bld version.

xmlCopy

<groupId>com.svenruppert</groupId><artifactId>core</artifactId><version>0.0.1-SNAPSHOT</version>

Start by creating a newbld build script in Java.bld uses pure Java code to define the build logic, dependencies, and configurations, allowing you to migrate the settings from the**pom.xml**. First, create a**Build.java** file (or similar) in your**src/bld/java/** directory, if not already existing. Here is a starting point for the migration:

javaCopy

packagecom.svenruppert;importrife.bld.Project;importrife.bld.dependencies.VersionNumber;importjava.nio.file.Path;importjava.util.List;publicclassCoreBuildextendsProject{ publicCoreBuild(){   pkg="com.svenruppert";   name="Core";   mainClass="com.svenruppert.Application";   version=version(0,1,0,"SNAPSHOT"); publicstaticvoidmain(String[]args){   newCoreBuild().start(args); }}

As next we are migrating the definition of the repositories.

xmlCopy

<repositories> <repository>   <snapshots>     <enabled>false</enabled>   </snapshots>   <id>central</id>   <name>libs-release</name>   <url>https://repo.maven.apache.org/maven2/</url> </repository> <repository>   <snapshots>     <enabled>true</enabled>     <updatePolicy>always</updatePolicy>   </snapshots>   <id>snapshots</id>   <name>libs-snapshot</name>   <url>https://repo.maven.apache.org/maven2/</url> </repository></repositories>

This will lead to a statement like the following inside the Build-Class.

javaCopy

repositories=List._of_(_MAVEN_CENTRAL_,_RIFE2_RELEASES_);

For the most projects you will need some dependencies in different scopes. The xml - Version wil look like the following. I am not listing all the dependencies of the demo project.

xmlCopy

<dependency> <groupId>io.javalin</groupId> <artifactId>javalin</artifactId> <version>6.3.0</version></dependency><dependency> <groupId>io.javalin</groupId> <artifactId>javalin-testtools</artifactId> <version>6.3.0</version> <scope>test</scope></dependency>

The bld Version:

javaCopy

VersionNumberversionJavalin=version(6,3,0);scope(compile)   .include(dependency("io.javalin","javalin",versionJavalin));scope(test)   .include(dependency("io.javalin","javalin-testtools",versionJavalin));

Finally we a have to migrate the plugins from maven to bld. If your Maven project used custom lifecycle phases, you can createcustom commands inbld using Java methods annotated with**@BuildCommand**. This allows you to replicate any special build steps that were part of your Maven process, like packaging, publishing, or precompilation tasks.

Example of adding a custom command inbld :

javaCopy

@BuildCommandpublicvoidcustomTask(){    System.out.println("Running custom task...");}

Here we are at a point, where we can see, that bld is not as feature ritch as maven out of the box. But, if you need support for another plugin or lifecycle, have a look at the documentation about extensions. Here you will find all you need to create one by yourself. The cool thing here is, that everything can be done on core java. Even complex tasks can be implemented including all technologies that are needed for this step.

But, back to the migration. In my case I am using thepitest plugin to generate the mutation test coverage reports.

Inside thepom.xml you have to declare the following:

xmlCopy

<plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <version>1.14.1</version> <dependencies>   <dependency>     <groupId>org.pitest</groupId>     <artifactId>pitest-junit5-plugin</artifactId>     <version>1.2.1</version>   </dependency> </dependencies> <configuration>   <threads>2</threads>   <outputFormats>     <outputFormat>XML</outputFormat>     <outputFormat>HTML</outputFormat>   </outputFormats>   <targetClasses>     <param>${pitest-prod-classes}</param>     <!--<param>org.reflections.*</param>-->   </targetClasses>   <targetTests>     <param>${pitest-test-classes}</param>   </targetTests> </configuration></plugin>

The correspondingbld part will be:

javaCopy

scope(test)   .include(dependency("org.pitest","pitest",version(1,17,0)))   .include(dependency("org.pitest","pitest-command-line",version(1,17,0)))   .include(dependency("org.pitest","pitest-junit5-plugin",version(1,2,1)))   .include(dependency("org.pitest","pitest-testng-plugin",version(1,0,0)));@BuildCommand(summary="Run PIT mutation tests")publicvoidpit()throwsException{ newPitestOperation()     .fromProject(this)     .reportDir(Path.of("reports","mutations").toString())     .targetClasses(pkg+".*")     .targetTests("junit."+pkg+".*")     .verbose(true)     .execute();}

Now, you are able to trigger the pitest extension on commandline with./bld pitest. To activate the plugin there must be the declaration inside the file “lib/bld/bld-wrapper.properties ”. The value of this key is the plugin maven coordinates including the version number.

propertiesCopy

bld.extensions=com.uwyn.rife2:bld-pitest:1.0.2

With this we are able to migrate maven project into bld-projects.

Conclusion

Migrating a complex Maven project tobld involves translating the dependency management, build logic, and testing configurations frompom.xml into Java code usingbld ’s project class structure. While both tools manage dependencies and build processes,bld allows developers to write build logic in Java, giving more flexibility and making it easier to manage as part of the application codebase.

Withbld , you gain direct control over the build process through Java code, but you must ensure that all elements of your Maven build (dependencies, plugins, phases) are correctly transferred.

Happy Coding

Sven

]]>JavaToolsUncategorizedhttps://svenruppert.com/posts/what-are-component-based-web-application-frameworks-for-java-developers/Wed, 25 Sep 2024 17:04:18 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/what-are-component-based-web-application-frameworks-for-java-developers/Tapestry, Wicket, and Vaadin A component-oriented Java web application framework is a development framework that enables the construction of web applications in a modular way, using reusable, encapsulated components that manage their state, behaviour, and presentation. This approach allows developers to build complex user interfaces by assembling pre-built or custom components, like building blocks, each handling specific functionalities within the application.<![CDATA[

Tapestry, Wicket, and Vaadin

A component-oriented Java web application framework is a development framework that enables the construction of web applications in a modular way, using reusable, encapsulated components that manage their state, behaviour, and presentation. This approach allows developers to build complex user interfaces by assembling pre-built or custom components, like building blocks, each handling specific functionalities within the application.

1. Tapestry, Wicket, and Vaadin
2. What is the challenge in creating component-oriented web frameworks in Java?
   1. State Management
   2. Component Lifecycle
   3. Event Handling
   4. Performance Optimisation
   5. Seamless Integration with Client-Side Technologies
   6. Security Concerns
   7. Usability and Developer Productivity
   8. Scalability
3. What is Apache Tapestry?
   1. Use Cases
4. What is Apache Wicket?
   1. Key Features of Apache Wicket
   2. Use Cases
5. Vaadin Plattform
   1. Vaadin Flow
6. The Difference between Apache Tapestry, Apache Wicket and Vaadin Flow
   1. Apache Tapestry
   2. Apache Wicket
   3. Vaadin Flow
   4. Key Differentiators

Encapsulation : Each component is a self-contained unit that includes its own logic and data handling. This encapsulation helps hide the internal details of components while exposing a defined interface with which other components can interact.

Reusability : Components are designed to be reused across different parts of an application or even across multiple projects. This can significantly speed up development and ensure consistency throughout the application.

Modularity : The application can be divided into smaller, manageable parts (components), which can be developed, tested, and maintained independently. This modularity makes it easier to manage complex applications and allows multiple developers to work on different components simultaneously without conflicts.

Interactivity : Component-oriented frameworks often support event-driven programming, allowing components to react to user inputs or other system events. This makes it straightforward to build dynamic and interactive web applications.

Integration : These frameworks typically provide easy integration with other technologies, such as databases, web services, and third-party libraries, allowing components to interact with these resources smoothly.

What is the challenge in creating component-oriented web frameworks in Java?

Creating component-oriented web frameworks in Java poses several challenges, primarily due to the complexities of web development and the nature of Java as a server-side technology. Here are some of the main challenges:

State Management

Web applications are inherently stateless, meaning each request is independent of previous requests. Component-oriented frameworks need to manage the state across multiple requests and user sessions. This can be complex, particularly when considering the need to synchronise the state between the server and the client while ensuring performance and scalability.

Component Lifecycle

Managing the lifecycle of components—creation, initialisation, rendering, updates, and destruction—is a significant challenge. Each component must handle its lifecycle efficiently, interacting with other components and the broader application environment without memory leaks or performance bottlenecks.

Event Handling

In a typical desktop application, the event-driven model is straightforward because the application runs in a single environment. However, in web applications, the user interface runs in the browser (client-side), while the business logic often runs on the server. This separation requires a robust mechanism to handle events on the client side, process them on the server, and reflect the changes to the client, all without making the application feel sluggish to the user.

Performance Optimisation

Component-oriented frameworks can introduce overhead due to the need for fine-grained components to manage their state and behaviour. This can impact performance, as every user interaction might require server communication. Handling this communication and minimising the data transferred between client and server is crucial for maintaining a responsive user experience.

Seamless Integration with Client-Side Technologies

While Java is used on the server side, the client side typically involves HTML, CSS, and JavaScript. Ensuring that Java components integrate seamlessly with these technologies and that the framework can support rich client-side interactions is challenging. This includes providing support for Ajax and JavaScript callbacks and updating UI parts without refreshing the entire page.

Security Concerns

Component-oriented frameworks need to manage security at the component level, ensuring that components do not expose vulnerabilities (such as cross-site scripting (XSS) or cross-site request forgery (CSRF)) and that the application can securely manage user data and authentication across components.

Usability and Developer Productivity

The framework should be robust, flexible, and easy to use. Providing a high level of abstraction while allowing developers enough control to customise component behaviour and appearance is a delicate balance. Documentation, tooling, and community support are crucial for developer adoption and productivity.

Scalability

As web applications grow, they must efficiently handle increasing loads and concurrent users. Component-oriented frameworks must ensure that components do not become bottlenecks and that the application can scale horizontally across multiple servers or instances.

Addressing these challenges requires a thoughtful design of the framework’s architecture, focusing on efficiency, flexibility, and ease of use to make the development of complex web applications feasible and efficient.

What is Apache Tapestry?

Apache Tapestry is an open-source component-based web application framework for Java. It is designed to simplify the development of complex web applications by providing a highly scalable, efficient, and expressive programming model. Tapestry focuses on high productivity and maintainable code, adhering to the “convention over configuration” principle, which minimises the need for extensive XML configuration files commonly seen in older Java frameworks.

Component-Based Architecture : Tapestry structures applications as a collection of components. Each component has its template (typically an HTML file) and an optional Java class that manages its behaviour. This model encourages reusable UI segments and separation of concerns.

Live Class Reloading : One of Tapestry’s standout features is its ability to reload Java classes automatically whenever they are changed. This means developers can modify the code and see the effects immediately in the browser without restarting the server, enhancing developer productivity.

Convention Over Configuration : Tapestry reduces the need for explicit configuration by following convention over configuration. For example, it automatically detects templates based on naming conventions. This approach minimises boilerplate and setup code, making projects easier to manage and quicker to develop.

High Performance : Tapestry includes built-in support for aggressive caching and minimisation of server-side processing, which enhances the application’s performance. It generates compact, efficient JavaScript code and CSS, optimising applications’ load time and runtime performance.

Inbuilt Ajax Support : Tapestry supports Ajax, allowing developers to quickly update parts of the page and manage client-server communication asynchronously. This is handled in a way that abstracts much of the complexity associated with Ajax in raw JavaScript.

Powerful Templating : Each Tapestry component can have its HTML template, allowing designers and developers to work together seamlessly. The framework automatically links HTML templates with their corresponding Java classes, providing a robust foundation for clean, maintainable code.

Dependency Injection and IoC Container : Tapestry integrates a powerful Inversion of Control (IoC) container that manages the lifecycle of application objects, centralising configuration and promoting loose coupling. This IoC container also supports dependency injection, simplifying the management of dependencies within the application.

Use Cases

Tapestry is particularly well-suited for developers looking for a Java-based framework that supports rapid development cycles and needs a scalable solution for enterprise-grade applications. Its strong support for AJAX and client-side interactions makes it suitable for applications requiring rich, dynamic user interfaces.

What is Apache Wicket?

Apache Wicket, often called Wicket, is a component-based web application framework for the Java programming language. It distinguishes itself with a strong emphasis on simplicity, ease of use, and a clean separation between logic and markup. Wicket allows developers to write applications in pure Java using components without the need for extensive XML configuration files.

Key Features of Apache Wicket

Component-Based Architecture : Wicket is built around a component-based model that treats pages and their elements as reusable components. These components are backed by Java classes and associated HTML templates. This design enables the encapsulation of behaviour and promotes reuse.

Pure HTML Templates : Wicket uses pure HTML for its templates, allowing designers to create and edit them without needing to understand or embed any particular syntax or placeholders. Java components dynamically bind to these HTML templates via Wicket-specific attributes, keeping the development clean and straightforward.

Strong Separation of Concerns : The framework promotes a clear separation between markup (HTML) and business logic (Java), making it easier for designers and developers to collaborate. Changes to the page layout do not typically require changes in the Java code.

Event-driven : Components in Wicket are event-driven, similar to Java Swing or AWT. They handle events through callback methods, simplifying the management of user interactions and partial page updates.

Stateful or Stateless : Wicket supports stateful components (preserving state across requests) and stateless components (where each request is treated as a new interaction), giving developers flexibility based on their application requirements.

AJAX Support : The framework has built-in support for AJAX, allowing developers to update parts of a web page asynchronously without refreshing the entire page. This seamless integration, with Wicket managing much of the complexity behind the scenes.

Security Features : Wicket is designed with security in mind, providing features to prevent common vulnerabilities such as Cross-Site Scripting (XSS) and Cross-Site Request Forgery (CSRF). These protections are built into the framework and enabled by default.

Testability : The Wicket was designed to be testable at the unit and integration levels. It provides mock objects and testing mechanisms to simulate user interaction and application flow without requiring a running web server.

Use Cases

Wicket is suitable for developers who prefer a pure Java approach to web development without relying heavily on JavaScript. It’s particularly appealing in environments where application maintainability and a clear separation between code and design are priorities. This makes it ideal for enterprise applications where long-term maintainability and scalability are critical.

Vaadin Plattform

The Vaadin Platform is a comprehensive development suite that simplifies building modern web applications. It is especially favoured for its ability to enable the creation of user interfaces (UIs) using Java, a language widely used across various industries, particularly in enterprise settings. Here are some key components and features of the Vaadin Platform:

Vaadin Flow : This is the core framework for building web UIs in Java. Flow lets developers manage the UI and its interactions on the server side, automating the communication between the client (browser) and server.

Vaadin Components : Vaadin comes with a rich set of built-in components, such as buttons, text fields, grids, and charts, ready to use out of the box. These components are built on web components, making them modern and highly interactive.

Vaadin Fusion : Targeted at developers who prefer client-side development, Fusion allows building web applications using TypeScript and leveraging the same backend APIs that Flow uses. This supports a more traditional approach to web development using JavaScript/TypeScript for the frontend while maintaining Java for the backend.

Themes and Layouts : Vaadin supports customisable themes and layouts, allowing developers to tailor the appearance of their applications to match specific branding or aesthetic requirements. It uses standard CSS for styling.

Tools and Integrations : Vaadin provides various tools and integrations for popular IDEs (like Eclipse, IntelliJ IDEA, and VS Code), build tools (such as Maven and Gradle), and frameworks (like Spring). This integration helps streamline the development process, making it more efficient.

End-to-End Development : From front-end components to back-end integration, Vaadin aims to provide a full-stack development experience. It supports building applications with responsive design, accessibility features, and internationalisation.

Community and Pro Add-ons : The Vaadin platform also offers a range of community-driven and professional add-ons that extend the functionality of the basic framework. These can include advanced data components, tools for enhanced productivity, and integrations with other systems.

Vaadin Flow

Vaadin Flow is a Java framework for building modern web applications. Flow is specifically designed to let developers build web UIs in pure Java without needing HTML or JavaScript, though these can still be used if desired.

Here’s how Vaadin Flow works:

Server-Side Java : The application’s logic is written entirely in Java. Vaadin Flow automatically manages the communication between the client-side (browser) and the server-side.

Components and Layouts : Flow provides a range of UI components (like buttons, grids, and forms) and layouts that are used to construct the application interface.

Data Binding : It supports robust data binding capabilities, making it easy to connect UI components to data sources and synchronising data between the server and the client.

Routing and Navigation : Vaadin Flow handles routing and navigation within the application, managing different views that users interact with.

Vaadin Flow is ideal for developers familiar with Java and prefer to work in a server-side programming environment. It allows them to create rich, interactive web applications while remaining trendy in enterprise environments where Java is a common choice for backend development.

The Difference between Apache Tapestry, Apache Wicket and Vaadin Flow

Apache Tapestry, Apache Wicket, and Vaadin Flow are all Java-based frameworks designed to facilitate the development of web applications through a component-oriented approach. However, they each have distinct characteristics, philosophies, and design implementations. Here’s a detailed comparison:

Apache Tapestry

Component Model : Tapestry uses a model where pages and components are Java objects that are mapped to templates defined in HTML files. Each component is highly reusable and can be embedded within other components.

Templates and Scripting : Tapestry firmly separates Java code from the HTML templates, promoting clean architecture and allowing designers to work on HTML/CSS without needing to understand Java.

Conventions over Configuration : Tapestry relies heavily on convention over configuration, reducing the need for XML configuration files and aiming for zero configuration where possible.

Live Class Reloading : One of Tapestry’s hallmark features is its live class reloading capability, which allows developers to change code and see results immediately without restarting the server.

Performance : Tapestry includes built-in support for JavaScript and AJAX without requiring deep developer knowledge, automatically optimising client-server communication.

Apache Wicket

Component Model : Wicket treats web pages as a hierarchy of components, each backed by its own Java class and usually associated with an HTML template that defines its layout.

State Management : Wicket is known for being a stateful framework by default, which means it automatically manages the state across requests, but it can also be configured to be stateless.

HTML and Java Integration : Wicket allows dynamic content to be written in pure HTML, with placeholders linked to Java components. This enables a clean separation between the presentation layer and business logic.

Event Model : It follows an event-driven model similar to desktop GUI programming, making it intuitive for developers with experience in such environments.

Testability : Wicket is designed to be highly testable with support for mocking and easy unit testing of individual components.

Vaadin Flow

Component Model : Vaadin Flow allows developers to build their UI entirely in Java or combine Java and templates. Vaadin 14 introduced LitTemplate, which lets developers define components using TypeScript and LitHTML.

Client-Server Communication : Flow abstracts the client-server communication completely, allowing developers to focus on server-side Java without worrying about the frontend, which is automatically handled.

Rich Widget Set : Vaadin has a comprehensive set of highly interactive UI components that resemble desktop functionality.

Themes and Layouts : It supports powerful theming capabilities with CSS and theming constructs that are easy to apply across different components.

Performance and Scalability : Vaadin manages client-server communication efficiently but can be challenged by the large amount of data sent over the network, especially in complex applications with numerous UI components.

Key Differentiators

Approach to HTML/Java Separation : Tapestry and Wicket encourage more direct involvement with HTML, facilitating collaboration between designers and developers. Vaadin, in contrast, allows developers to work almost entirely in Java, which might streamline development but can distance the design process from HTML/CSS.

State Management : Wicket is inherently stateful, which is beneficial for certain applications but can impact performance and scalability. Tapestry and Vaadin manage the state more transparently and can be more scalable in large applications.

Ease of Use and Learning Curve : Vaadin is often considered easier to learn for Java developers since it requires less HTML and JavaScript knowledge. Tapestry and Wicket have steeper learning curves but offer more control and flexibility over the UI.

Live Reloading : Tapestry’s live class reloading is particularly useful for rapid development cycles, a feature not inherently present in Wicket or Vaadin.

Choosing between these frameworks depends mainly on the project requirements, team skills, and specific needs regarding application scalability, maintainability, and development speed. Each framework has strengths and can be the best choice in different scenarios.

]]>JavaVaadinhttps://svenruppert.com/posts/cwe-1123-excessive-use-of-self-modifying-code-for-java-developers/Thu, 12 Sep 2024 11:19:19 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cwe-1123-excessive-use-of-self-modifying-code-for-java-developers/Self-modifying code refers to a type of code that alters its own instructions while it is executing. While this practice can offer certain advantages, such as optimisation and adaptability, it is generally discouraged due to the significant risks and challenges it introduces. For Java developers, using self-modifying code is particularly problematic because it undermines the codebase’s predictability, readability, and maintainability, and Java as a language does not natively support self-modification of its code.<![CDATA[

Self-modifying code refers to a type of code that alters its own instructions while it is executing. While this practice can offer certain advantages, such as optimisation and adaptability, it is generally discouraged due to the significant risks and challenges it introduces. For Java developers, using self-modifying code is particularly problematic because it undermines the codebase’s predictability, readability, and maintainability, and Java as a language does not natively support self-modification of its code.

1. Risks
2. Examples of Risky Practices
   1. Example
   2. Mitigation Strategies
3. Example
   1. Mitigation Strategies
4. Mitigation Strategies
5. CVE-2014-0114
6. CVE-2013-2423
7. CVE-2015-1832
8. CVE-2012-0507
9. CVE-2019-12384
10. Mitigation Strategies
11. Code Injection Attacks
12. Remote Code Execution (RCE)
13. Privilege Escalation
14. Dynamic Code Loading Attacks
15. Polymorphic Malware
16. Evasion of Security Mechanisms
17. Backdoors and Rootkits
18. Tampering with Security Features

Risks

Unpredictable Behavior: Self-modifying code can lead to unexpected program behaviour, making diagnosing and fixing bugs difficult.

Security Vulnerabilities: Code that modifies itself can be a vector for various security attacks, including injection attacks and malware.

Maintenance Difficulty: Such code is difficult to read and understand, making it more difficult to maintain and update.

Performance Issues: Self-modifying code can cause performance degradation due to the additional overhead of modifying and interpreting the changes at runtime.

Examples of Risky Practices

Dynamic Class Loading: Java allows classes to be loaded at runtime using mechanisms such as reflection or custom class loaders. While dynamic class loading itself is not inherently wrong, using it excessively or without apparent necessity can lead to self-modifying behaviour.

Bytecode Manipulation: Using libraries like ASM or Javassist to modify Java bytecode at runtime can lead to self-modifying code. This practice is highly discouraged unless essential.

Reflection: While reflection is a powerful feature, it can be misused to modify private fields, methods, or classes, leading to behaviour that is hard to trace and debug.

Example

An example of risky self-modifying behaviour in Java using bytecode manipulation:

javaCopy

importjavassist.ClassPool;importjavassist.CtClass;importjavassist.CtMethod;publicclassSelfModifyingExample{publicstaticvoidmain(String[]args){try{ClassPoolpool=ClassPool.getDefault();CtClasscc=pool.get("TargetClass");CtMethodm=cc.getDeclaredMethod("targetMethod");m.insertBefore("{ System.out.println(\"Method modified at runtime\"); }");cc.toClass();TargetClasstarget=newTargetClass();target.targetMethod();}catch(Exceptione){e.printStackTrace();}}}classTargetClass{publicvoidtargetMethod(){System.out.println("Original method execution");}}

In this example, the TargetClass method targetMethod is modified at runtime to include an additional print statement. This kind of modification can lead to the aforementioned risks.

Mitigation Strategies

Avoid Runtime Code Modifications: Design your system in a way that minimises or eliminates the need for runtime code modifications.

Use Design Patterns: Employ design patterns such as Strategy or State patterns that allow behaviour changes without altering the code at runtime.

Proper Use of Reflection: Use reflection sparingly and only when no other viable solution exists. Document its usage thoroughly.

Static Code Analysis: Use static code analysis tools to detect and prevent the introduction of self-modifying code.

Excessive use of self-modifying code in Java is fraught with risks that can compromise your applications’ security, maintainability, and performance. By adhering to best practices and using design patterns that promote flexibility and adaptability without modifying code at runtime, you can avoid the pitfalls associated with CWE-1123.

A Reflection Example

Reflection in Java allows for introspection and manipulation of classes, fields, methods, and constructors at runtime. While powerful, excessive or improper use of reflection can lead to self-modifying behaviours, which aligns with CWE-1123. This can result in unpredictable behaviour, security vulnerabilities, and maintenance challenges.

Example

Below is an example demonstrating the excessive use of reflection to modify a class’s behaviour at runtime, which can be considered a form of self-modifying code.

javaCopy

importjava.lang.reflect.Field;importjava.lang.reflect.Method;publicclassReflectionExample{publicstaticvoidmain(String[]args){try{// Original object creationMyClassoriginal=newMyClass();original.printMessage();// Using reflection to modify the behavior at runtimeClass<?>clazz=Class.forName("MyClass");Methodmethod=clazz.getDeclaredMethod("setMessage",String.class);method.setAccessible(true);// Modify the private field value using reflectionFieldfield=clazz.getDeclaredField("message");field.setAccessible(true);field.set(original,"Modified message");// Verify the modificationoriginal.printMessage();}catch(Exceptione){e.printStackTrace();}}}classMyClass{privateStringmessage="Original message";publicvoidprintMessage(){System.out.println(message);}privatevoidsetMessage(Stringmessage){this.message=message;}}

In this example, the ReflectionExample class:

Creates an instance of MyClass and prints the original message. It uses reflection to modify the private field message and the private method setMessage of MyClass. Changes the value of the message field and prints the modified message.

This example showcases how reflection can alter an object’s behaviour and state at runtime, leading to the issues outlined in CWE-1123.

Mitigation Strategies

Minimise Reflection Use: Avoid using reflection unless absolutely necessary. Prefer alternative design patterns that allow for flexibility without modifying the code at runtime.

Access Control: Ensure that fields and methods that should not be modified are kept private and final where possible to prevent unintended access.

Static Analysis Tools: Use static analysis tools to detect excessive use of reflection and other risky practices in the codebase.

Code Reviews: Conduct thorough code reviews to identify and mitigate the use of self-modifying code through reflection.

Reflection is a powerful tool in Java, but misuse can lead to the risks associated with CWE-1123. By adhering to best practices and minimising the use of reflection to modify code at runtime, developers can maintain the security, predictability, and maintainability of their applications.

Example of Dynamic Class Loading

Dynamic class loading in Java refers to the ability to load and unload classes at runtime. While this can be useful in specific scenarios, excessive or improper use can lead to self-modifying code behaviours, which align with CWE-1123. This can introduce risks such as unpredictable behaviour, security vulnerabilities, and maintenance challenges.

Below is an example demonstrating the excessive use of dynamic class loading to modify a class’s behaviour at runtime, which can be considered a form of self-modifying code.

javaCopy

publicclassDynamicClassLoadingExample{publicstaticvoidmain(String[]args){try{// Load the original classClassLoaderclassLoader=DynamicClassLoadingExample.class.getClassLoader();Class<?>loadedClass=classLoader.loadClass("MyClass");// Create an instance of the loaded classObjectinstance=loadedClass.getDeclaredConstructor().newInstance();// Invoke the original methodloadedClass.getMethod("printMessage").invoke(instance);// Dynamically load the modified classclassLoader=newCustomClassLoader();loadedClass=classLoader.loadClass("ModifiedClass");// Create an instance of the modified classinstance=loadedClass.getDeclaredConstructor().newInstance();// Invoke the modified methodloadedClass.getMethod("printMessage").invoke(instance);}catch(Exceptione){e.printStackTrace();}}}// Original class definitionclassMyClass{publicvoidprintMessage(){System.out.println("Original message");}}// Custom class loader to simulate loading a modified classclassCustomClassLoaderextendsClassLoader{@OverridepublicClass<?>loadClass(Stringname)throwsClassNotFoundException{if("ModifiedClass".equals(name)){// Define a modified version of MyClass at runtimeStringmodifiedClassName="ModifiedClass";StringmodifiedClassBody="public class "+modifiedClassName+" {"+" public void printMessage() {"+" System.out.println(\"Modified message\");"+" }"+"}";byte[]classData=compileClass(modifiedClassName,modifiedClassBody);returndefineClass(modifiedClassName,classData,0,classData.length);}returnsuper.loadClass(name);}privatebyte[]compileClass(StringclassName,StringclassBody){// Simulate compiling the class body into bytecode (in a real scenario, use a compiler API)// This is a placeholder for demonstration purposesreturnclassBody.getBytes();}}

In this example, the DynamicClassLoadingExample class:

Loads an original class MyClass and invokes its printMessage method. Dynamically loads a modified version of the class, ModifiedClass, using a custom class loader. Creates an instance of the modified class and invokes its printMessage method, which prints a different message.

This example showcases how dynamic class loading can alter a program’s behaviour at runtime, leading to the issues outlined in CWE-1123.

Mitigation Strategies

Avoid Unnecessary Dynamic Loading: Use dynamic class loading only when it is indispensable and cannot be avoided through other design patterns.

Secure Class Loaders: Ensure custom class loaders are secure and do not load untrusted or malicious classes.

Static Analysis Tools: Use static analysis tools to detect excessive use of dynamic class loading and other risky practices in the codebase.

Code Reviews: Conduct thorough code reviews to identify and mitigate the use of self-modifying code through dynamic class loading.

Java-based CVEs based on CWE-1123

While there might not be specific CVEs (Common Vulnerabilities and Exposures) explicitly labelled as being caused by CWE-1123 (Excessive Use of Self-Modifying Code), several Java-related vulnerabilities can arise from practices associated with self-modifying code. These typically involve dynamic class loading, reflection, and bytecode manipulation issues. Here are some examples of Java-based CVEs that relate to these practices:

CVE-2014-0114

Description: Apache Commons Collections Remote Code Execution Vulnerability

Issue: This vulnerability involves using reflection to manipulate serialised data, leading to arbitrary code execution. It was found in the Apache Commons Collections library, where certain classes could be used to execute arbitrary code when deserialised. This is a form of self-modifying behaviour, as the serialised data could alter the program’s execution flow.

Impact: Attackers could exploit this vulnerability to execute arbitrary commands on the server running the vulnerable application.

CVE-2013-2423

Description: Oracle Java SE Remote Code Execution Vulnerability

Issue: This vulnerability arises from improper handling of certain methods in Java, leading to the execution of arbitrary code. It leverages reflection and class-loading mechanisms to inject and execute malicious code.

Impact: Exploiting this vulnerability allows remote attackers to execute arbitrary code on the affected system, potentially leading to total system compromise.

CVE-2015-1832

Description: Android Remote Code Execution Vulnerability in Apache Cordova

Issue: This vulnerability involves dynamic class loading and improper validation of inputs. It allowed attackers to inject malicious code into an Android application built with Apache Cordova by exploiting the WebView component.

Impact: Successful exploitation could result in arbitrary code execution within the context of the affected application, leading to potential data leakage or further exploitation.

CVE-2012-0507

Description: Oracle Java SE Remote Code Execution Vulnerability

Issue: This vulnerability involves using reflection and dynamic class loading to exploit a flaw in the Java Runtime Environment (JRE). The vulnerability allows an untrusted Java applet to break out of the Java sandbox and execute arbitrary code.

Impact: Exploiting this vulnerability could allow an attacker to execute arbitrary code on the host system with the user’s privileges running the Java applet.

CVE-2019-12384

Description: FasterXML jackson-databind Deserialization Vulnerability

Issue: This vulnerability involves the unsafe handling of deserialisation using the jackson-databind library. By exploiting polymorphic type handling, attackers could inject malicious code that gets executed during deserialisation.

Impact: Successful exploitation could result in arbitrary code execution, leading to potential data breaches and system compromise.

Mitigation Strategies

Avoid Self-Modifying Code Practices: Do not use dynamic class loading, reflection, or bytecode manipulation unless absolutely necessary. When required, ensure proper validation and security measures are in place.

Use Safe Deserialisation: Avoid deserialisation of untrusted data. If deserialisation is necessary, libraries and techniques that enforce strict type checking and validation should be used.

Apply Security Patches: Regularly update and patch libraries and frameworks to protect against known vulnerabilities.

Code Reviews and Static Analysis: Conduct thorough code reviews and use static analysis tools to detect and mitigate the use of risky code practices.

Security Best Practices: To reduce the attack surface, follow security best practices, such as least privilege, input validation, and secure coding guidelines.

What kind of attacks or infection methods are based on CWE-1123?

CWE-1123 (Excessive Use of Self-Modifying Code) can lead to several types of attacks and infection methods due to such code’s unpredictable and dynamic nature. Here are some common attack vectors and infection methods associated with this vulnerability:

Code Injection Attacks

Attackers exploit self-modifying code to inject malicious code into a program. This can occur through various means, such as manipulating input data that gets executed or modifying code at runtime to include harmful payloads.

Example:

SQL Injection: If an application dynamically constructs SQL queries using user input and modifies these queries at runtime, an attacker can inject malicious SQL commands to alter the behaviour of the database operations.

Remote Code Execution (RCE)

Self-modifying code can enable Remote Code Execution by allowing attackers to modify or load classes and methods at runtime. This makes it easier to introduce and execute arbitrary code.

Example:

Deserialization Vulnerabilities: When an application deserialises data without proper validation, an attacker can inject objects that modify the code flow, leading to the execution of arbitrary code.

Privilege Escalation

Attackers can exploit self-modifying code to escalate their privileges within a system. They can bypass security checks and gain higher-level access by dynamically altering the code.

Example:

Reflection Attacks: Using reflection, attackers can modify private fields and methods to escalate privileges, accessing parts of the system that would otherwise be restricted.

Dynamic Code Loading Attacks

Self-modifying code often involves dynamic loading of classes or bytecode manipulation, which can be exploited to load malicious code at runtime.

Example:

Dynamic Class Loading: Attackers can trick the application into loading a malicious class that performs unwanted actions, such as exfiltrating data or modifying system files.

Polymorphic Malware

Self-modifying code is commonly used in polymorphic malware, where the malware changes its code to evade detection by security software.

Example:

Polymorphic Virus: A virus that encrypts its payload and changes its decryption routine with each infection, making it difficult for antivirus programs to detect the malware’s signature.

Evasion of Security Mechanisms

Self-modifying code can be used to evade security mechanisms such as firewalls, intrusion detection systems (IDS), and antivirus software by altering its code structure dynamically.

Example:

Metamorphic Malware: Similar to polymorphic malware, metamorphic malware reprograms itself completely with each infection, ensuring that no two copies of the malware are identical, thus evading signature-based detection.

Backdoors and Rootkits

Attackers can use self-modifying code to install backdoors or rootkits that alter the behaviour of the operating system or application to provide persistent unauthorised access.

Example:

Rootkits: A rootkit can use self-modifying code to hide its presence by altering kernel or application code to prevent detection by security tools.

Tampering with Security Features

Self-modifying code can be used to tamper with security features such as authentication mechanisms, encryption routines, and access controls.

Example:

Tampering with Authentication: By dynamically modifying authentication checks, an attacker can bypass login mechanisms and gain unauthorised access to the system.

By understanding these attack vectors and implementing mitigation strategies, developers and security professionals can reduce the risks associated with self-modifying code and improve the overall security of their applications.

Happy Coding

Sven

]]>JavaSecure Coding PracticesSecurityhttps://svenruppert.com/posts/iot-with-tinkerforge-and-java/Wed, 04 Sep 2024 12:37:30 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/iot-with-tinkerforge-and-java/Introduction TinkerForge is an innovative and modular hardware system that allows users to build electronic devices quickly and flexibly. Whether you’re an experienced engineer, a hobbyist, or a complete newbie, TinkerForge offers a range of components that can be easily connected and programmed, allowing for rapid prototyping and the creation of custom electronics projects. Since its launch, TinkerForge has gained popularity in various areas, including education, research, and industrial automation, due to its user-friendly design and extensive feature set.<![CDATA[

Introduction

TinkerForge is an innovative and modular hardware system that allows users to build electronic devices quickly and flexibly. Whether you’re an experienced engineer, a hobbyist, or a complete newbie, TinkerForge offers a range of components that can be easily connected and programmed, allowing for rapid prototyping and the creation of custom electronics projects. Since its launch, TinkerForge has gained popularity in various areas, including education, research, and industrial automation, due to its user-friendly design and extensive feature set.

1. Introduction
2. The origins and evolution of TinkerForge
3. TinkerForge architecture
   1. Bricks
   2. Bricklets
   3. Master-Extensions
4. Programming and control
   1. TinkerForge-API
   2. TinkerForge-GUI
   3. Remote access and control
5. Applications of TinkerForge
   1. Training
   2. Prototyping
   3. Industrial automation
   4. DIY projects and maker community
6. Advantages of TinkerForge
7. Limitations and challenges
8. How do I start programming TinkerForge electronics with Java?
   1. Requirements
   2. TinkerForge architecture
   3. Connecting your TinkerForge hardware
      1. Connect Bricklets to the Master Brick
      2. Connect the master brick to the computer.
   4. Installing the TinkerForge Java API
   5. Writing the first Java program
      1. Running and testing the program
      2. Dealing with errors
      3. Event-driven programming
      4. Controlling actors
   6. Conclusion

The origins and evolution of TinkerForge

TinkerForge was founded in 2011. The founders Matthias Bolte and Olaf Lüke imagined a system that would lower the hurdles in hardware development. Their goal was to provide a range of components that could be easily combined into complex systems without requiring in-depth knowledge of electronics or soldering skills.

The idea arose from the frustration the founders experienced while working on electronics projects, which often required a lot of time just to get the essential components to work together. They recognised that there was a need for a more intuitive and modular system that would allow anyone to create hardware projects quickly and effectively.

Over the years, TinkerForge has grown from a niche product into a widely used platform supported by a vibrant community of developers and manufacturers. The system is now used in various applications, from simple DIY projects to advanced industrial automation systems.

TinkerForge architecture

The TinkerForge system is based on a modular architecture consisting of three main component types: Bricks, Bricklets and Master Extensions.

Bricks

Bricks are the core building blocks of the TinkerForge system. They are small, self-contained modules that provide specific functions, such as controlling motors, reading sensors, or connecting to a computer. Each brick has a specific purpose and can be easily attached using a stackable system.

The most common types of bricks include:

Master Brick : The Master Brick is the central node that controls and supplies power to other Bricks and Bricklets. It provides the communication interface between the TinkerForge stack and a computer or microcontroller.

ESP32 Brick : The ESP32 Brick offers sixBricklet Connections and is equipped with a powerful ESP32 microcontroller.

HAT-Brick : With the HAT Brick up to act, bricklets can be connected to a Raspberry Pi.

IMU-Brick 2.0 : The Inertial Measurement Unit Brick contains a 9 DOF (Degrees of Freedom) sensor that provides orientation data, making it suitable for navigation and motion-tracking applications.

Bricklets

Bricklets are smaller modules that connect to bricks to expand their functionality. They are essentially sensors, actuators, or other peripheral devices that can be connected to a brick to provide additional input or output options. Bricklets are connected to Bricks via standardised connectors, and each can support multiple Bricklets.

Some examples of Bricklets are:

Temperature Bricklet : This module measures temperature and can be used in projects requiring environmental monitoring.

Distance IR Bricklet : This sensor measures the distance to an object using infrared light, which is helpful in robotics or object detection applications.

LCD 20x4 Bricklet : A simple display module that enables text output, perfect for building user interfaces or displaying sensor data.

Master-Extensions

Master Extensions are additional boards that can be connected to the master brick to provide additional functionality or expand connectivity. These extensions allow the TinkerForge system to connect to various communication protocols or external devices.

Examples of master extensions are:

WIFI extension 2.0 : This module adds wireless communication capabilities to the TinkerForge system, allowing bricks to be controlled remotely via a computer or smartphone.

RS485 extension : This extension enables long-distance communication via RS485, making it suitable for industrial environments where long-distance wired connections are required.

Ethernet extension : This extension provides a wired Ethernet connection to the TinkerForge system, ensuring reliable and fast communication in networked environments.

Programming and control

One of TinkerForge’s outstanding features is its ease of programming. The system is designed to be accessible to users of all experience levels. TinkerForge offers various ways to control and program the Bricks and Bricklets, ensuring users can choose the best method.

TinkerForge-API

TinkerForge provides a powerful and easy-to-use application programming interface (API) that allows users to control the Bricks and Bricklets from various programming languages ​​, including Python, C, C++, Java, Ruby and more. The API abstracts the complexity of hardware interaction and simplifies writing code that interacts with the various components.

The API is well-documented and includes examples for each supported language. This documentation provides detailed explanations of each feature, making it easier for users to understand how to control the hardware.

https://www.tinkerforge.com/en/doc/Software/API_Bindings_Java.html

TinkerForge-GUI

TinkerForge also offers a graphical user interface (GUI) called Brick Viewer for users who prefer a more visual approach. This software visually represents the connected Bricks and Bricklets and allows users to configure, monitor and control their devices without writing code.

Brick Viewer is handy for beginners who are just starting with the TinkerForge system. It allows users to experiment with the hardware, view sensor data in real-time, and adjust settings without understanding the underlying code.

https://www.tinkerforge.com/de/doc/Software/Brickv.html#brickv

Remote access and control

TinkerForge also supports remote access and control, allowing users to interact with their devices over a network. The WIFI or Ethernet Master extensions allow users to control their TinkerForge systems over the Internet from anywhere in the world. This feature is precious for projects that require remote monitoring or control, such as home automation systems or remote sensing applications.

Applications of TinkerForge

TinkerForge’s versatility has led to its use in various applications. The most common areas of use for TinkerForge include:

Training

TinkerForge is an excellent tool for electronics and programming lessons. Its modular design allows students to experiment with different components and see immediate results. The ease of use and extensive documentation make it accessible to learners of all ages.

Educational institutions often use TinkerForge in robotics, embedded systems, and computer science courses. It allows students to create projects demonstrating key concepts such as sensor integration, motor control and data communication.

Prototyping

One of TinkerForge’s strengths is its ability to enable rapid prototyping. Engineers and designers can quickly assemble and test their ideas without complex wiring or soldering. The system’s modular design means that components can be easily replaced or reconfigured as needed.

TinkerForge is often used in the early stages of product development, where speed and flexibility are crucial. Using TinkerForge, developers can iterate their designs faster, reducing the time to market for new products.

Industrial automation

TinkerForge is also used in industrial automation applications where reliability and scalability are important. The system is suitable for use in industrial environments by supporting various communication protocols, such as RS485 and Ethernet.

In these environments, TinkerForge components can monitor and control machines, collect data from sensors, and automate processes. The modular design allows for easy expansion and customisation, making building complex systems tailored to specific requirements possible.

DIY projects and maker community

TinkerForge has a strong following among DIYers and hobbyists. The system’s simplicity and flexibility make it ideal for creating custom electronics projects, such as home automation systems, robotics, and art installations.

The maker community around TinkerForge is active, and users share their projects, code and ideas online. This community-focused approach has led to the development of a wide range of open-source projects and resources, further increasing the value of the TinkerForge ecosystem.

Advantages of TinkerForge

TinkerForge offers several advantages that have contributed to its popularity:

Modularity : The ability to mix and match components allows users to create highly customised systems without designing everything from scratch.

Ease of use : With a straightforward API and visual interface, TinkerForge is accessible to users of all experience levels.

Scalability : TinkerForge can be used in small, simple projects and large, complex systems. Its modular design allows it to be easily expanded as needed.

Cross-platform support : TinkerForge’s API is available in multiple programming languages, making it easy to integrate into existing projects.

Community and support : A strong community and comprehensive documentation ensure users have access to the resources they need to succeed.

Limitations and challenges

Although TinkerForge is a powerful and flexible system, it also has its limitations:

Costs : Although the price is reasonable, building complex systems with TinkerForge can be expensive, especially when multiple bricks and brackets are required.

Limited computing power : Some bricks, especially the master brick, may not have the computing power required for demanding applications. In such cases, an external microcontroller or computer may be required.

Learning curve : Although TinkerForge is designed to be user-friendly, there is still a learning curve, especially for those new to electronics or programming.

Dependence on proprietary hardware : TinkerForge components are proprietary, meaning users are somewhat locked into the ecosystem. While the open API and documentation help mitigate this, it is worth considering for long-term projects.

TinkerForge represents a significant advancement in the field of modular electronics, providing a platform that bridges the gap between hardware and software. Its user-friendly design, modular architecture, and extensive documentation make it an ideal choice for a wide range of users.

How do I start programming TinkerForge electronics with Java?

Requirements

Before diving into Java programming with TinkerForge, you must have the following:

Basic knowledge of Java : Familiarity with Java syntax and programming concepts.

TinkerForge-Hardware : At least oneMeister Brick and aBricklet (e.g. Temperature Bricklet) for interaction.

Computer with USB port : To connect the master brick to the computer.

Internet connection : To download the required software and libraries.

TinkerForge architecture

Before programming, it is essential to understand the vital components of the TinkerForge system:

Master Bricks : The central controller that connects to your computer via USB. It manages communication with all connected Bricklets.

Bricklets : Modular components that fulfil specific functions (e.g. sensors, actuators). They are connected to the master brick via standardised connectors.

Stackable System : Multiple bricks and bricklets can be stacked to create complex systems without soldering.

Connecting your TinkerForge hardware

Connect Bricklets to the Master Brick

First, the Bricklets should be connected to the Master Brick. Care should be taken to ensure that the connection is tight and that the plugs’ pins are aligned correctly.

Example setup :

• Master Brick: Connected via USB.
• Temperature Bricklet: Connected to one of the ports of the master brick.

Connect the master brick to the computer.

After connecting the master brick to the computer using a USB cable, it should boot up, as indicated by the LEDs lighting up.

Installing the TinkerForge Java API

To interact with TinkerForge hardware via Java, you need the TinkerForge Java API. This example uses Maven to manage the dependencies. The coordinates for the TinkerForge dependencies are:

xmlCopy

<dependency>    <groupId>com.tinkerforge</groupId>    <artifactId>tinkerforge</artifactId>    <version>2.1.33</version></dependency>

Writing the first Java program

The following example shows a simple Java program that reads temperature data from the Temperature Bricklet. For this purpose, a Java class called “TemperatureReader ` created. The following imports are necessary for this program. Typically, the IDE should automatically display the relevant suggestions.

javaCopy

importcom.tinkerforge.BrickletTemperature;importcom.tinkerforge.IPConnection;importcom.tinkerforge.TinkerforgeException;

The temperature bracket’s UID is required below. The UID is usually printed on the bricklet or can be found via the TinkerForge Brick Viewer.

javaCopy

publicclassTemperatureReader{    privatestaticfinalStringHOST="localhost";    privatestaticfinalintPORT=4223;// Replace XYZ with your Bricklet UID    privatestaticfinalStringTEMPERATURE_BRICKLET_UID="XYZ";}

For simplicity, the actual program is implemented directly in the main method of the class.

javaCopy

publicstaticvoidmain(String[]args){    IPConnectionipcon=newIPConnection();// Create IP connection// Create device object    BrickletTemperaturetemp=newBrickletTemperature(TEMPERATURE_BRICKLET_UID,ipcon);    try{        ipcon.connect(HOST,PORT);// Connect to brickd        System.out.println("Connected to TinkerForge");        // Read temperature        shorttemperature=temp.getTemperature();        System.out.println("Temperature: "+temperature/100.0+" °C");        ipcon.disconnect();    }catch(TinkerforgeExceptione){        System.err.println("Error: "+e.getMessage());        e.printStackTrace();    }}

note : Replace “XYZ” with the actual UID of the temperature bricklet.

IP connection : Manages the connection to the TinkerForge daemon (**brickd**), which facilitates communication between the computer and the bricks/bricklets.

BrickletTemperature : Represents the temperature bricklet element and allows interaction with its functions.

getTemperature(): This function gets the current temperature in hundredths of a degree Celsius. The value is converted to degrees Celsius by dividing by 100.0.

Running and testing the program

Start the TinkerForge daemon (“brickd”).

Before running the Java program, make sure the TinkerForge daemon is running. This daemon manages communication between the computer and the TinkerForge hardware.

Running the Java program

After starting the methodTemperatureReader.main() ’’ you can see the following output on the console.

Connected to TinkerForge

Temperature: 23,45 °C

Of course, the temperature value varies depending on the environment.

Dealing with errors

If errors occur, note the following:

Incorrect UID : Make sure the UID in the source code matches the UID of the Temperature Bricklet.

Daemon is not running : Check if the**brickd** is running and connected to the master brick.

Connection problems : Check the USB connections and ensure the master brick is properly connected to the computer.

Event-driven programming

Now that a simple Java program for reading temperature data has been created, advanced features can be explored. Instead of actively querying temperature data, callbacks can be set up to receive data asynchronously.

javaCopy

importcom.tinkerforge.BrickletTemperature;importcom.tinkerforge.IPConnection;importcom.tinkerforge.TinkerforgeException;publicclassTemperatureCallback{    privatestaticfinalStringHOST="localhost";    privatestaticfinalintPORT=4223;    privatestaticfinalStringTEMPERATURE_BRICKLET_UID="XYZ";// Replace XYZ with your Bricklet UID    publicstaticvoidmain(String[]args){        IPConnectionipcon=newIPConnection();// Create IP connection// Create device object        BrickletTemperaturetemp=newBrickletTemperature(TEMPERATURE_BRICKLET_UID,ipcon);        try{            ipcon.connect(HOST,PORT);// Connect to brickd            System.out.println("Connected to TinkerForge");            // Register temperature callback            temp.addTemperatureListener((temperature)->{                System.out.println("Temperature Callback: "+temperature/100.0+" °C");            });            // Set callback period to 1 second (1000 ms)            temp.setTemperatureCallbackPeriod(1000);            // Keep the program running            System.out.println("Press Ctrl+C to exit.");            while(true){                Thread.sleep(1000);            }        }catch(TinkerforgeException|InterruptedExceptione){            System.err.println("Error: "+e.getMessage());            e.printStackTrace();        }    }}

Explanation:

addTemperatureListener : Registers a callback function that fires every time the temperature is updated.

setTemperatureCallbackPeriod : Defines the interval (in milliseconds) at which the callback is invoked.

Controlling actors

If actuators such as LEDs or motors have been connected via Bricklets, they can be controlled programmatically.

Example: Controlling an LED bricklet

javaCopy

importcom.tinkerforge.BrickletLED;importcom.tinkerforge.IPConnection;importcom.tinkerforge.TinkerforgeException;publicclassLEDControl{    privatestaticfinalStringHOST="localhost";    privatestaticfinalintPORT=4223;// Replace ABC with your LED Bricklet UID    privatestaticfinalStringLED_BRICKLET_UID="ABC";    publicstaticvoidmain(String[]args){        IPConnectionipcon=newIPConnection();// Create IP connection        BrickletLEDled=newBrickletLED(LED_BRICKLET_UID,ipcon);// Create device object        try{            ipcon.connect(HOST,PORT);// Connect to brickd            System.out.println("Connected to TinkerForge");            // Turn LED on            led.setColorLEDValue("on");            System.out.println("LED is ON");            // Wait for 2 seconds            Thread.sleep(2000);            // Turn LED off            led.setColorLEDValue("off");            System.out.println("LED is OFF");            ipcon.disconnect();        }catch(TinkerforgeException|InterruptedExceptione){            System.err.println("Error: "+e.getMessage());            e.printStackTrace();        }    }}

Conclusion

Programming with Java on the TinkerForge platform provides a powerful way to create interactive and intelligent electronic projects. There are hardly any limits to your imagination here. Thanks to the connection to Java, you can link your projects with many other technologies, which dramatically simplifies the integration of existing infrastructures and projects.

Happy coding and creating!

]]>JavaTinkerForgehttps://svenruppert.com/posts/building-more-complex-apps-with-flow/Thu, 22 Aug 2024 12:20:26 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/building-more-complex-apps-with-flow/In this part of the series about Vaadin Flow, I will show how I can create the basic framework for the graphic design of a work application. The focus here is on the design of the work area and the organisation of the individual logical application groups. In other words, we create the application layout that can be used for an industrial project.<![CDATA[

In this part of the series about Vaadin Flow, I will show how I can create the basic framework for the graphic design of a work application. The focus here is on the design of the work area and the organisation of the individual logical application groups. In other words, we create the application layout that can be used for an industrial project.

Of course, this also includes dealing with different languages ​​based on i18N.

One of the first steps in an application is often to think about how everything should be presented on the screen. If menus are used, headers and footers should be used for elements, and what about the basic design of the workspace? The questions here are to be understood as examples, as they will, of course, always depend on the context and the target group.

1. AppLayout
2. What is i18N - Internationalization in Java
3. i18N in Vaadin Flow
4. Conclusion

AppLayout

AppLayout is a reasonably robust and easy-to-use solution that comes with the Vaadin Flow framework. This class provides the entry point for quickly building an application’s framework. We looked at this class briefly in the first part and will now go into more detail here.

A classicApplayout consists of a header, where part of the navigation is usually housed. There is also a corresponding footer, which is usually used for lower-priority status messages. The middle is then divided into one to three parts. We will choose a classic two-column design here. The left side is the main navigation, the detailed navigation of which will then be visible in the header like a menu bar.

The layout consists of three sections: a horizontal navigation bar (called Navbar), a fold-out navigation bar (drawer), and a content area. An application’s main navigation blocks should be positioned in the navigation bar and/or drawer while rendering views in the content area.

The app layout is responsive and automatically adapts to the screen size of desktops, tablets, and mobile devices. This behaviour saves us a lot of customisation work that we would otherwise have to implement ourselves.

The navigation structure depends mainly on the number of elements to be shown and whether a sub-navigation is required.

The navigation bar is a good choice for multiple items (3-5) as they fit in the viewport without scrolling. If you need to display more items or small screen support is a priority, the Drawer is a better choice. It can accommodate a longer list of links without scrolling and collapses into a hamburger menu on small screens. Additionally, a vertical list of items is easier for the user to scan. If multi-level or hierarchical navigation is required, Draver is the first level. The secondary (and tertiary) navigation elements can be placed using the drawer or in the navigation bar (Navbar).

The implementation can then look like this.

We create the basic framework inMainLayout from the classAppLayout is derived. Here, we have to use the methods for setting the individual navigation areas. Let’s start by building the main navigation, which is on the left side in this example. Here you will see a combination of icon and descriptions for the main areas of the application. The necessary graphical elements are displayed along with the application title via the methodaddToDrawer(…) set.

The application title is an instance of the classH1. Here the text to be displayed is simply selected using the methodsetText() handover.

The navigation elements are created using the classSideNavItem realised. The transfer parameters are a label. a target view and an icon. AllSideNavItem - Instances are in the instance of the classSideNav summarised and transferred to the layout. In case the number of entries is larger and you want the user to be able to scroll through this list, the instance of the classSideNav is embedded into an instance of the classScroller.

In the source code, it looks like this:

javaCopy

privateScrollerprimaryNavigation(){    SideNavsideNav=newSideNav();    sideNav.addItem(        newSideNavItem("Dashboard",DashboardView.class,            VaadinIcon.DASHBOARD.create()),        newSideNavItem("Orders",AllOrdersView.class,            VaadinIcon.CART.create()),        newSideNavItem("Customers",CustomersView.class,            VaadinIcon.USER_HEART.create()),        newSideNavItem("Products",ProductsView.class,            VaadinIcon.PACKAGE.create()),        newSideNavItem("Documents",DocumentsView.class,            VaadinIcon.RECORDS.create()),        newSideNavItem("Tasks",TasksView.class,            VaadinIcon.LIST.create()),        newSideNavItem("Analytics",AnalyticsView.class,            VaadinIcon.CHART.create()));    Scrollerscroller=newScroller(pageNav);    scroller.setClassName(LumoUtility.Padding.SMALL);    returnscroller;  }

To complete the layout, we will put all the elements together.

This is implemented in the constructor.

javaCopy

privateH1appTitle=newH1();  publicMainLayout(){    addToDrawer(appTitle,primaryNavigation());    setPrimarySection(Section.DRAWER);  }

In this example, you can see that the labels are all stored directly in the source code. We will change this later so that we can make the application multilingual. I ask for a bit of patience here. We also see the specification of the targets using the class name. We’ll look at that a little later, too.

In the previous post, I showed how the MainLayout class is used.

Screen with the created menu items

If you want to create a sub-navigation, you can make additional menu entries in the header on the right. This is often used when you want to divide a somewhat more complex process into different views. For example, let’s take the illustration for dealing with orders. Let’s just assume that there are the sub-areas “All orders ”, “Open orders ”, “Completed orders " and “Abandoned orders ” must give. This results in another four navigation destinations, which are, however, summarised under the item Orders.

Screenshot after navigating to the Orders menu item

These submenu items are the same for all views that deal with orders. This immediately raises the question of how to extract this menu. Great mechanisms from inheritance come into play here. We create a classAbstractHeaderView andAbstractOrdersView. All constructs required for a generic implementation of secondary navigation are stored in the first class mentioned. This includes, for example, how the second navigation bar should be positioned, what graphic properties it has and whether there is a hamburger symbol that can be used to show or hide the main navigation.

Ultimately, all that needs to be added is which navigation points must be created for each implementation. The respective implementation of the methodsecondaryNavigationsLinks() defines these menu items.

The following listing shows the implementation for the screenshot shown, divided into the generic partAbstractHeaderView and the specific one for the ordersAbstractOrdersView.

xmlCopy

public abstract class AbstractViewHeader    extends Composite<VerticalLayout> {  public AbstractViewHeader(String subTitle) {    initView(subTitle);  }  protected RouterLink createLink(String viewName,                                  Class<? extends Composite> viewClass) {    RouterLink link = new RouterLink();    link.add(viewName);    link.setRoute(viewClass);    link.addClassNames(LumoUtility.Display.FLEX,        LumoUtility.AlignItems.CENTER,        LumoUtility.Padding.Horizontal.MEDIUM,        LumoUtility.TextColor.SECONDARY,        LumoUtility.FontWeight.MEDIUM);    link.getStyle().set("text-decoration", "none");    return link;  }  public void initView(String subTitle) {    DrawerToggle toggle = new DrawerToggle();    H2 viewTitle = new H2(getTranslation(subTitle));    HorizontalLayout titleBar = new HorizontalLayout(toggle, viewTitle);    titleBar.setAlignItems(FlexComponent.Alignment.CENTER);    titleBar.setSpacing(false);    VerticalLayout viewHeader = getContent();    viewHeader.add(titleBar, secondaryNavigation());    viewHeader.setPadding(false);    viewHeader.setSpacing(false);  }  private HorizontalLayout secondaryNavigation() {    HorizontalLayout navigation = new HorizontalLayout();    navigation.addClassNames(        LumoUtility.JustifyContent.CENTER,        LumoUtility.Gap.SMALL,        LumoUtility.Height.MEDIUM);    secondaryNavigationLinks().forEach(navigation::add);    return navigation;  }  public abstract List<RouterLink> secondaryNavigationLinks();public abstract class AbstractOrdersView    extends AbstractView<HorizontalLayout>    implements LocaleChangeObserver {  public static final String CANCELLED = "orders.subnavigation.cancelled";  public static final String COMPLETED = "orders.subnavigation.completed";  public static final String OPEN = "orders.subnavigation.open";  public static final String ALL = "orders.subnavigation.all";  private final String subTitle;  public AbstractOrdersView(String subTitle) {    this.subTitle = subTitle;  }  @Override  protected AbstractViewHeader createViewHeader() {    return new AbstractViewHeader(subTitle) {      @Override      public List<RouterLink> secondaryNavigationLinks() {        return List.of(            createLink(getTranslation(ALL), AllOrdersView.class),             createLink(getTranslation(OPEN), OpenOrdersView.class),             createLink(getTranslation(COMPLETED), CompletedOrdersView.class),             createLink(getTranslation(CANCELLED), CancelledOrdersView.class));      }    };  }}

Here, you can clearly see that the source text is greatly reduced when there are a large number of navigation elements. A minimal basic framework is created to implement the variations for the individual views. I have intentionally refrained from including functions for processing orders in this example because I want to focus on the structure here.

javaCopy

@Route(value="allOrders",layout=MainLayout.class)@PreserveOnRefreshpublicclassAllOrdersViewextendsAbstractOrdersView{  publicstaticfinalStringSUB_TITLE="orders.all.subtitle";  publicAllOrdersView(){    super(SUB_TITLE);     //the necessary elements are placed here    getContent().add(newTextField("TBD ALL Orders"));  }  @Override  publicvoidlocaleChange(LocaleChangeEventevent){    // fuer i18N  }}

With the basic framework shown here, you can build quite complex navigations. Now, let’s get to the topic of i18n. A web application usually needs to support multiple languages. I will not discuss languages ​​here that require a change in reading direction. In this example, I’ll focus on supporting any number of languages ​​with a left-to-right reading order. Examples of languages ​​here will be German and English. The primary function here is quite simple

What is i18N - Internationalization in Java

The basic concept ofi18n (internationalisation) in Java is to design and develop an application that can be easily adapted to different languages ​​and regions without requiring significant changes to the code.

Resource bundle

A resource bundle is a set of key-value pairs that store country-specific data, such as B. Strings for messages, labels and other UI components. In Java, resource bundles are usually property files named after the locale they represent (e.g. “messages_en_US.properties " for US English, “messages_de.properties “ for general German).

When the application runs, it dynamically loads the appropriate resource bundle based on the user’s locale. This allows the application to display content in the user’s preferred language without hard-coding the text in the source code.

Locale

Alocale in Java is an object that represents a specific geographical, political or cultural region. It includes language, country and sometimes variant codes. For example,Locale.ENGLISH represents the English language andLocale.US represents the United States.

The class**Local** customises information for the user, such as B. Text, dates, numbers and currencies, according to its region.

Message Formatting

Java provides classes like**MessageFormat** to handle complex message formatting that allows dynamic insertion of values ​​into localised messages (e.g. “Welcome, {0}!”, where{0} can be replaced with the user’s name).

You define message templates in your resource bundles, and at runtime, the class replaces**MessageFormat** Placeholders with actual values ​​to ensure messages are appropriately localised and formatted.

Date, number and currency formatting

Dates, numbers, and currencies are often represented differently in different locales (e.g.MM/DD/YYYY vs.DD/MM/YYYY for dates). Java provides classes like**DateFormat**,**NumberFormat** and**Currency** to process these deviations.

These classes format data according to locale settings and ensure users see information in the expected format.

Fallback mechanism

If a specific locale’s resource package is unavailable, Java uses a fallback mechanism to select a more general package (e.g., falling back frommessages_fr_CA.properties tomessages_fr.properties).

This ensures that the application remains functional even if a full set of localised resources is unavailable for each locale.

The basic concept of i18n in Java is to create applications that can support multiple languages ​​and regional settings with minimal changes to the code base. This includes using resource bundles, locale-dependent classes for formatting, and a fallback mechanism to ensure a seamless user experience across different locales.

i18N in Vaadin Flow

To use i18N, the application only needs to have the corresponding text files with the keys and contents available in the classpath under the vaadin-i18n directory with the filename prefix translations (e.g. src/main/resources/vaadin-i18n/translations.properties).

When a translation is requested or when the I18NProvider is used for the first time, the vaadin-i18n directory is checked to see if it contains any translations.properties or translations_[langcode].properties files. All language codes are collected from the available files and added as provided locales in the DefaultI18NProvider.

The translations.properties file is a standard translation file used for any locale that does not have a specific translation file. For example, locale translation files are called translations_de.properties or translations_en.properties. Automatic locale creation supports one to three parts (e.g. translations_language_country_variant.properties).

The locale to use is determined by matching the locales provided by the I18NProvider with the Accept-Language header in the client’s initial response. It will be used if an exact match (i.e., language and country) is found. Otherwise, it will just try to find a match by language. If neither is found, the locale defaults to the first “supported” locale fromI18NProvider.getProvidedLocales() set. If this is empty, willLocale.getDefault() used.

Implementing internationalisation in an application is a combination of usingI18NProvider and updating translations when locale changes. This can be done, for example, by the user if he wants to change the language. To simplify this, the application classes that control the localised captions and text canLocaleChangeObserver implement to receive events related to locale changes. During navigation, this observer will also be notified when the component is attached, but beforeonAttach() is called. All URL parameters from navigation are set so that they can be used to determine status.

javaCopy

publicclassLocaleObserverextendsDivimplementsLocaleChangeObserver{    @Override    publicvoidlocaleChange(LocaleChangeEventevent){        setText(getTranslation("my.translation",getUserId()));    }}

If you want to set the texts explicitly, you can use this method:getTranslation(..).

javaCopy

publicclassMyLocaleextendsDiv{    publicMyLocale(){        setText(getTranslation("my.translation",getUserId()));    }}

In this example project, the listing for creating the main menu entries is rewritten. For clarity, here are both versions for a menu entry.

High Version :

sqlCopy

newSideNavItem("Orders",AllOrdersView.class,            VaadinIcon.CART.create())

New version :

javaCopy

publicstaticfinalStringMENU_ITEM_ORDERS="mainlayout.menuitem.orders";newSideNavItem(            getTranslation(MENU_ITEM_ORDERS),            AllOrdersView.class,            CART.create())

translation.properties :

mainlayout.menuitem.orders=Orders

Conclusion

With these two elements, the ApplLayout and the realisation or implementation of i18N, two fundamental building blocks for developing a web application have been laid. From here, the application can now be built step by step. We will deal with the individual topics in the next parts, so it remains exciting.

Happy Coding

Sven

]]>JavaVaadinhttps://svenruppert.com/posts/cwe-377-insecure-temporary-file-in-java/Wed, 21 Aug 2024 13:17:12 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cwe-377-insecure-temporary-file-in-java/In software development, temporary files are often used to store data temporarily during an application’s execution. These files may contain sensitive information or be used to hold data that must be processed or passed between different parts of a program. However, if these temporary files are not managed securely, they can introduce vulnerabilities that may compromise the application’s confidentiality, integrity, or availability. The Common Weakness Enumeration (CWE) identified CWE-377 as a weakness associated with the insecure creation and management of temporary files.<![CDATA[

In software development, temporary files are often used to store data temporarily during an application’s execution. These files may contain sensitive information or be used to hold data that must be processed or passed between different parts of a program. However, if these temporary files are not managed securely, they can introduce vulnerabilities that may compromise the application’s confidentiality, integrity, or availability. The Common Weakness Enumeration (CWE) identified CWE-377 as a weakness associated with the insecure creation and management of temporary files.

1. Understanding CWE-377
   1. CWE-377 in Java
   2. Example of Vulnerable Code
   3. Potential Impacts
   4. Mitigations
      1. UseFile.createTempFile() Properly
      2. Ensure Proper File Permissions
      3. Avoid Hardcoded File Names
      4. Handle Temporary Files in a Privileged Context
   5. Advanced Considerations
      1. Usejava.nio.file Package
      2. Consider Using In-Memory Solutions
   6. Best Practices
2. OpenSource Libraries for secure temp file handling
   1. Apache Commons IO
   2. Google Guava
   3. JUnit 5 (JUnit Jupiter)
3. Practical examples
   1. RESTful API Demo - Upload with Javalin
   2. Insecure Implementation
   3. Secure Implementation
   4. Key Differences
   5. Conclusion on CWE-377: Insecure Temporary File

Understanding CWE-377

CWE-377: Insecure Temporary File refers to a security weakness that occurs when a program creates a temporary file in an insecure manner. Attackers can exploit this vulnerability to perform various malicious activities, including data tampering, unauthorised data access, or denial of service. Insecure temporary file creation typically involves issues such as:

Predictable file names : If temporary files have predictable names, attackers can guess the file names and either read or modify the file contents.

Insecure file permissions : Incorrect permissions can allow unauthorised users to access or modify temporary files.

Race conditions : A time-of-check to time-of-use (TOCTOU) race condition may occur if an attacker creates a file with the same name as the one the application intends to generate before the application does.

CWE-377 in Java

In Java, developers often use temporary files for various purposes, such as caching, processing intermediate data, or storing temporary results. Java provides several ways to create temporary files, such as the**File.createTempFile()** method, which generates a temporary file with a unique name in the default temporary-file directory. However, improper use of these APIs can lead to CWE-377.

Example of Vulnerable Code

Consider the following example:

javaCopy

importjava.io.File;importjava.io.IOException;publicclassInsecureTempFileExample{    publicstaticvoidmain(String[]args)throwsIOException{        FiletempFile=newFile("/tmp/tempfile.txt");        tempFile.createNewFile();        System.out.println("Temporary file created at: "+tempFile.getAbsolutePath());    }}

In this example, the code creates a temporary file with a hardcoded file name (tempfile.txt) in the/tmp directory. This approach is insecure for several reasons: The file name is predictable, allowing an attacker to create a file with the same name before the application does, leading to a TOCTOU race condition. The file is created without any control over file permissions, potentially exposing sensitive data.

Potential Impacts

CWE-377 can have severe consequences depending on how the temporary file is used and the sensitivity of the data it contains. Some potential impacts include:

Information Disclosure : If an attacker can predict the name of a temporary file, they may be able to read its contents if the file is not adequately protected. This can lead to disclosing sensitive information, such as passwords, tokens, or personal data.

Data Tampering : An attacker could create or modify a temporary file before the application uses it, resulting in data corruption or unauthorised modifications. This could lead to incorrect application behaviour or the introduction of malicious data into the system.

Denial of Service (DoS) : By preemptively creating temporary files with the names an application expects to use, an attacker can prevent the application from functioning correctly, leading to a denial of service.

Mitigations

To prevent CWE-377, developers must adhere to secure coding practices when working with temporary files. Below are several mitigation strategies for securely creating and managing temporary files in Java:

UseFile.createTempFile() Properly

TheFile.createTempFile() method generates a unique temporary file name, reducing the risk of predictable file names. It also allows developers to specify a directory for the file, though it defaults to the system’s temporary-file directory.

javaCopy

importjava.io.File;importjava.io.IOException;publicclassSecureTempFileExample{    publicstaticvoidmain(String[]args)throwsIOException{        FiletempFile=File.createTempFile("tempfile_",".tmp");        tempFile.deleteOnExit();// Ensures the file is deleted when the JVM exits        System.out.println("Temporary file created at: "+tempFile.getAbsolutePath());    }}

This approach mitigates several risks:

• The file name is generated randomly, making it difficult for an attacker to predict.
• The file is automatically deleted when the Java Virtual Machine (JVM) exits, reducing the likelihood of stale files.

Ensure Proper File Permissions

When creating temporary files, setting appropriate file permissions is crucial to prevent unauthorised access. In Java, you can use the**setReadable()**,**setWritable()**, and**setExecutable()** methods to control file permissions.

javaCopy

importjava.io.File;importjava.io.IOException;publicclassSecureTempFileWithPermissionsExample{    publicstaticvoidmain(String[]args)throwsIOException{        FiletempFile=File.createTempFile("secure_tempfile_",".tmp");        tempFile.setReadable(true,true);        tempFile.setWritable(true,true);        tempFile.setExecutable(false);        tempFile.deleteOnExit();        System.out.println("Temporary file created with secure permissions at: "+tempFile.getAbsolutePath());    }}

In this example, the file is readable and writable only by the owner, minimising the risk of unauthorised access.

Avoid Hardcoded File Names

Using hardcoded file names, as seen in the initial insecure example, is risky because it makes the temporary file name predictable. Always use mechanisms that generate unique, unpredictable file names, such as**File.createTempFile()**.

Handle Temporary Files in a Privileged Context

Managing temporary files in a more controlled or privileged environment may be beneficial when dealing with sensitive data. This could involve creating the files in a directory with restricted access or using Java’s**AccessController** to enforce stricter security policies around file operations.

Advanced Considerations

Usejava.nio.file Package

The**java.nio.file** package, introduced in Java 7, provides more robust and flexible mechanisms for file handling, including temporary file creation. The**Files** class offers the**createTempFile()** method, which can also specify file attributes like permissions.

xmlCopy

import java.nio.file.Files;import java.nio.file.Path;import java.nio.file.attribute.PosixFilePermissions;import java.util.Set;public class NioSecureTempFileExample {    public static void main(String[] args) throws IOException {        Set<PosixFilePermission> permissions = PosixFilePermissions.fromString("rw-------");        Path tempFile = Files.createTempFile("nio_tempfile_",                                              ".tmp",                                             PosixFilePermissions.asFileAttribute(permissions));        System.out.println("Temporary file created with NIO at: " + tempFile.toAbsolutePath());    }}

This example shows how to set file permissions using the NIO package, providing fine-grained control over file security attributes.

Consider Using In-Memory Solutions

Some applications may be able to avoid creating temporary files altogether by using in-memory storage solutions, such as**ByteArrayOutputStream**, for temporary data. This approach eliminates the risks associated with file-system-based temporary storage but may not be suitable for large data sets or applications with significant memory constraints.

Best Practices

To ensure secure temporary file handling in Java, developers should adopt the following best practices:

Prefer Secure Defaults : Always use methods like**File.createTempFile()** or**Files.createTempFile()** that provide secure defaults for file creation.

Set File Permissions Explicitly : Ensure that temporary files have the minimal necessary permissions and avoid granting unnecessary access to other users.

Avoid Predictable File Names : Never use hardcoded or predictable names for temporary files. Always generate unique file names using secure APIs.

UsedeleteOnExit() : When possible, use**deleteOnExit()** to ensure that temporary files are cleaned up automatically when the JVM terminates.

Limit Scope of Temporary Files : Store temporary files in directories with restricted access, and consider creating a dedicated directory for temporary files that require stricter security controls.

Handle Exceptions Gracefully : Always handle exceptions when working with temporary files, ensuring the application remains secure and stable even if file operations fail.

CWE-377: Insecure Temporary File is a significant security concern in software development, particularly in sensitive data processing environments. In Java, developers must be vigilant in creating, managing, and securing temporary files to avoid introducing vulnerabilities that malicious actors could exploit.

By following the guidelines and best practices outlined in this document, Java developers can mitigate the risks associated with CWE-377 and ensure that their applications handle temporary files securely. The use of secure APIs, proper file permissions, and cautious file management practices are essential for maintaining the confidentiality, integrity, and availability of data within a Java application.

Ensuring that temporary files are managed securely is not only a best practice but a critical aspect of developing robust, secure Java applications. By adhering to these principles, developers can protect their applications from common threats and contribute to a safer software ecosystem.

OpenSource Libraries for secure temp file handling

Regarding secure temporary file handling in Java, several open-source libraries can help simplify and enhance security for managing temporary files. Here are some notable ones:

Apache Commons IO

Apache Commons IO provides a utility class for securely creating and managing temporary files.

Maven Dependency :

xmlCopy

<dependency>    <groupId>commons-io</groupId>    <artifactId>commons-io</artifactId>    <version>2.11.0</version></dependency>

Example Usage :

javaCopy

importorg.apache.commons.io.FileUtils;importjava.io.File;importjava.io.IOException;publicclassSecureTempFileExample{    publicstaticvoidmain(String[]args){        try{            // Create a secure temporary file            FiletempFile=               FileUtils.getTempDirectory()                        .createTempFile("secureTempFile",".txt");            tempFile.deleteOnExit();// Ensure the file is deleted on exit            // Optional: Set specific file permissions (POSIX systems)            if(tempFile.exists()){                tempFile.setReadable(true,true);                tempFile.setWritable(true,true);                tempFile.setExecutable(false,false);            }            System.out.println("Temporary file created at: "                 +tempFile.getAbsolutePath());        }catch(IOExceptione){            e.printStackTrace();        }    }}

Google Guava

Google Guava also provides utility methods for working with temporary files.

Maven Dependency :

xmlCopy

<dependency>    <groupId>com.google.guava</groupId>    <artifactId>guava</artifactId>    <version>31.0.1-jre</version></dependency>

Example Usage :

javaCopy

importcom.google.common.io.Files;importjava.io.File;importjava.io.IOException;publicclassSecureTempFileExample{    publicstaticvoidmain(String[]args){        try{            // Create a secure temporary file            FiletempFile=Files.createTempDir();            tempFile.deleteOnExit();// Ensure the file is deleted on exit            // Optional: Set specific file permissions (POSIX systems)            if(tempFile.exists()){                tempFile.setReadable(true,true);                tempFile.setWritable(true,true);                tempFile.setExecutable(false,false);            }            System.out.println("Temporary file created at: "                 +tempFile.getAbsolutePath());        }catch(Exceptione){            e.printStackTrace();        }    }}

JUnit 5 (JUnit Jupiter)

JUnit 5 provides a TempDir extension for securely creating temporary directories and files for testing purposes.

Maven Dependency :

xmlCopy

<dependency>    <groupId>org.junit.jupiter</groupId>    <artifactId>junit-jupiter-engine</artifactId>    <version>5.8.2</version></dependency>

Example Usage :

xmlCopy

import org.junit.jupiter.api.Test;import org.junit.jupiter.api.io.TempDir;import java.io.File;import java.io.IOException;import java.nio.file.Files;import java.nio.file.Path;public class SecureTempFileTest {    @TempDir    Path tempDir;    @Test    public void testCreateSecureTempFile() throws IOException {        // Create a secure temporary file in the provided temp directory        Path tempFile = Files.createTempFile(tempDir,                                              "secureTempFile",                                              ".txt");        // Optional: Set specific file permissions (POSIX systems)        Set<PosixFilePermission> permissions            = PosixFilePermissions.fromString("rw-------");        Files.setPosixFilePermissions(tempFile, permissions);        System.out.println("Temporary file created at: "             + tempFile.toAbsolutePath());    }}

Using these libraries can significantly simplify secure temporary file handling in Java applications. Apache Commons IO and Google Guava provide utilities for creating and managing temporary files, while JUnit 5 offers built-in support for creating temporary files and directories during testing.

Choose the library that best fits your needs and project dependencies. Apache Commons IO and Google Guava are versatile and widely used, whereas JUnit 5’s TempDir is excellent for secure temporary file handling in test cases.

Let’s discover some examples.

Practical examples

RESTful API Demo - Upload with Javalin

Javalin is a lightweight web framework for Java and Kotlin that can be used to create RESTful APIs and web applications. Below, I will provide two examples of handling file uploads and creating temporary files in Javalin: insecure and secure.

Insecure Implementation

In this insecure example, we will create temporary files with predictable names without setting appropriate permissions.

javaCopy

importio.javalin.Javalin;importjava.io.File;importjava.io.IOException;publicclassInsecureTempFileExample{    publicstaticvoidmain(String[]args){        Javalinapp=Javalin.create().start(7000);        app.post("/upload",ctx->{            // Save uploaded file to a temporary file             // with a predictable name            FileuploadedFile=ctx.uploadedFile("file").getContent();            FiletempFile=newFile("/tmp/insecure-temp-file.txt");            try{                uploadedFile.renameTo(tempFile);                ctx.result("File uploaded to: "                     +tempFile.getAbsolutePath());            }catch(Exceptione){                ctx.result("File upload failed");                e.printStackTrace();            }        });    }}

Secure Implementation

In this secure example, we will use Files.createTempFile to create temporary files with unique, unpredictable names and set appropriate file permissions.

xmlCopy

import io.javalin.Javalin;import java.io.InputStream;import java.nio.file.Files;import java.nio.file.Path;import java.nio.file.attribute.PosixFilePermission;import java.nio.file.attribute.PosixFilePermissions;import java.util.Set;public class SecureTempFileExample {    public static void main(String[] args) {        Javalin app = Javalin.create().start(7000);        app.post("/upload", ctx -> {            // Retrieve uploaded file            InputStream uploadedFile                 = ctx.uploadedFile("file").getContent();            // Define the prefix and suffix for the temporary file            String prefix = "secureTempFile";            String suffix = ".txt";            try {                // Create temporary file with default permissions                Path tempFile = Files.createTempFile(prefix, suffix);                // Optional: Set specific file permissions (POSIX systems)                Set<PosixFilePermission> permissions                    = PosixFilePermissions.fromString("rw-------");                Files.setPosixFilePermissions(tempFile, permissions);                // Write uploaded file content to temporary file                Files.copy(uploadedFile, tempFile);                ctx.result("File uploaded to: "                    + tempFile.toAbsolutePath().toString());            } catch (IOException e) {                ctx.result("File upload failed");                e.printStackTrace();            }        });    }}

Key Differences

1. File Creation :

sqlCopy

***Insecure**:Usesapredictablefilename(/tmp/insecure-temp-file.txt),whichcanleadtofilenamecollisionandunauthorisedaccess.***Secure**:UsesFiles.createTempFiletocreateaunique,unpredictablefilename.

2. File Permissions :

javaCopy

***Insecure**:Doesnotexplicitlysetfilepermissions,whichmayresultininsecuredefaultpermissions.***Secure**:Explicitlysetsrestrictivefilepermissions(rw-------),ensuringonlytheownercanreadandwritethefile.

3. Exception Handling :

javaCopy

***Bothexamples**:Handleexceptions,butthesecureexampleproperlyhandlesIOExceptionthatmayariseduringfileoperations.

By following the secure example, you can mitigate the risks associated with CWE-377 and ensure that temporary files are created securely in your Javalin applications.

Conclusion on CWE-377: Insecure Temporary File

CWE-377, concerning the insecure creation and management of temporary files, is a critical security vulnerability that can have far-reaching consequences if not adequately addressed. In the context of Java, this weakness often arises due to predictable file names, inadequate file permissions, and race conditions such as Time-of-Check to Time-of-Use (TOCTOU). These issues can lead to severe threats, including unauthorised access, data tampering, information leakage, and even denial of service.

To mitigate the risks associated with CWE-377, developers must adopt secure practices when handling temporary files. This includes using secure and atomic file creation methods, like**File.createTempFile()** and**Files.createTempFile()**, which generate unique and unpredictable file names while ensuring the atomicity of operations. Proper file permissions must also be enforced to limit access to temporary files, preventing unauthorised users from reading or modifying them.

Understanding the nuances of race conditions like TOCTOU is essential in designing secure applications. By eliminating the gap between checking a file’s existence and subsequent use, developers can prevent attackers from exploiting the system during that critical window.

In summary, addressing CWE-377 requires a combination of secure coding practices, careful file permissions management, and robust APIs that inherently mitigate common pitfalls associated with temporary files. By adhering to these practices, Java developers can protect their applications from the potentially severe impacts of insecure temporary file management, ultimately ensuring their software systems’ confidentiality, integrity, and availability.

]]>JavaSecure Coding PracticesSecurityhttps://svenruppert.com/posts/vaadin-flow-how-to-start/Wed, 19 Jun 2024 15:34:50 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/vaadin-flow-how-to-start/ We will now create a new Vaadin Flow application step by step and create a basic framework for our own projects with this component-based open-source web framework. So, right from the start, the question arises: How can you start with as little effort as possible without avoiding the usual expenses that sometimes come with creating projects?<![CDATA[

We will now create a new Vaadin Flow application step by step and create a basic framework for our own projects with this component-based open-source web framework. So, right from the start, the question arises: How can you start with as little effort as possible without avoiding the usual expenses that sometimes come with creating projects?

How and where do I start my project?

First we go to the URL start.vaadin.com. Here, we get to an application that allows us to generate a Vaadin project. Here, we choose the option “Hello World Project " out of. As soon as you get to the field “Edit ” on the right side with the mouse, the selection options shown here open. In this article, I will use Settingsa) Flow / Java,b) Java,c) Maven,d) Servlet.

I chose this setting specifically because I don’t want to have any additional technological dependencies on other frameworks at this point, making the project as small as possible.

After downloading the packed project, you can unpack it in a folder of your choice and get the complete directory structure necessary for a Maven project. To work with the project, please open the project with the IDE of your choice. I will use the IDE IntelliJ (https://www.jetbrains.com/idea/) from JetBrains in the community version in this article. This is freely available on all relevant platforms (Windows, Linux, and OSX). At this point I am assuming that a JDK and Maven are already installed and configured to work on the development computer.

After the project has been opened, you can do the first test to see whether everything is working. You can do this, for example, on the command line within the project directory with the following Maven command.mvn clean verify If everything runs through to the end without any error messages, you have downloaded all the necessary dependencies from the relevant repositories to your computer using Maven.

With the commandmvn clean package jetty:run, you can also try out the project straight away. Everything is translated, assembled, and started within the servlet container Jetty. If you now access the URL with a browser,http://localhost:8080/, you will see the Vaadin Flow application.

Now, you can try out how this application works. by entering something into the input field and then clicking the button labelled “Say hello ” clicks. In my case, I entered my name, “Sven Ruppert”, and activated the button once. The result then looks like this in my browser.

What is included in the project?

But how can you program this yourself? To do this, we first end the ongoing process that we are usingmvn have started.

First, we edit the pom.xml a little. Here, we will limit Vaadin’s dependency on pure open-source components. This saves time and space during development. For this purpose, the definition “vaadin ” changed in “vaadin-core ”

xmlCopy

<dependency>   <groupId>com.vaadin</groupId>   <artifactId>vaadin-core</artifactId></dependency>

The application logic

The example application consists of three Java classes. We’ll now start with the business logic we used in the class GreetingService find. There’s not much to say about this, as it’s a good old plain Java class. The goal is to transform the input string passed so that the greeting comes out. We don’t find any Vaadin elements here yet.

javaCopy

publicclassGreetService{   publicStringgreet(Stringname){       if(name==null||name.isEmpty()){           return"Hello anonymous user";       }else{           return"Hello "+name;       }   }}

Then, we are missing two more Java classes, which we can find in the source code folder. On the one hand, it’s the class ”MainView”. The entire visual design of this demo app is implemented here.

How do I define the GUI?

javaCopy

@Route("")publicclassMainViewextendsVerticalLayout{   publicMainView(){       TextFieldtextField            =newTextField("Your name");       GreetServicegreetService=newGreetService();       Buttonbutton           =newButton("Say hello",and->{               Stringvalue=textField.getValue();               Stringgreet=greetService.greet(value);               Paragraphparagraph=newParagraph(greet);               add(paragraph);           });       add(textField,button);   }}

Let’s go through this together. First, we see the class definition “MainView ”, which arises from deriving the class “VerticalLayout ” results. This is a layout component in which the elements are arranged vertically inside. Within the constructor, we now define the various visible elements. In this case, it is an input field and a button. At the very end, these are in the order of text field and button with the methodadd(..) added to the VerticalLayout. TheGreetingService is created within the constructor, and then within theActionListeners, the button is used. For this example, the procedure shown is exemplary. However, we will examine when you should use other design patterns in a later article. The button here has a clearly defined English label, “Say hello ”, that the user sees. Things get more interesting when defining behaviour. When you click the button, the values ​​are read from the text field using the instance ofGreetingServers modified. This new value is then embedded in a graphical component of the Paragraph type and passed to the external layout as a child node.

Here, you can see quite clearly that you don’t have to worry too much about the typical web application challenges during development. You can use everything that the Java language offers in terms of abstractions.

The MainView class has an annotation called “@Route(“”) ”. Die Annotation “@Route ” is used in Vaadin to map a view component (e.g. a UI) to a specific URL. It indicates which class is the web application’s main view for a particular path. This allows the user interface to be accessed via a specific URL in the browser.

The annotation’s value specifies the path under which the view can be reached. If no path is specified, the view is accessible at the root path ("/").

javaCopy

@Route("")   publicclassMainViewextendsVerticalLayout{       // View content   }

In this example, the**MainView** is under the root path (“http://localhost:8080/ “) reachable. You can also specify a specific path to make the view accessible at a specific URL.

javaCopy

@Route("all")   publicclassToDoViewextendsVerticalLayout{       // View content   }

The “ToDoView " under the path “http://localhost:8080/todo " is reachable in this example.

The annotation allows Vaadin to control navigation within the application. Users can navigate to different views by changing the URL in the browser or using programmatic navigation methods. The “@Route ” annotation can also be used with a layout. A layout is a high-level component that contains multiple views. Here is an example:

javaCopy

@Route(value="todo",layout=MainLayout.class)   publicclassToDoViewextendsVerticalLayout{       // View content   }

In this example, “ToDoView ” within the “MainLayout ” is displayed if the URL “http://localhost:8080/todo ” is called. You can also use dynamic parameters in the URL to display specific content in the view.

javaCopy

@Route("user/:userID")   publicclassUserViewextendsVerticalLayout{       // View content   }

In this example, “UserView ” at a URL like “http://localhost:8080/user/123 ” is called, whereas “123 ” is a dynamic parameter.

In summary, the**@Route** annotation defines URL mapping to View classes in Vaadin, which enables navigation and access to various parts of the web application.

How can I configure the web app?

Now, we’re missing the classAppShell , which is the implementation of the interfaceAppSehllConfigurator.

javaCopy

@PWA(name="Project Base for Vaadin",shortName="Project Base")@Push@Theme("my-theme")publicclassAppShellimplementsAppShellConfigurator{}

The interface ”AppShellConfigurator ” in Vaadin Flow defines the general configuration of the HTML shell of your Vaadin application. This mainly concerns the “< head>” area of ​​the HTML document where meta tags, titles, links to external resources (such as CSS files or favicons), and similar configurations can be set.

Here, for example, I added “Dark” to the theme definition, which now gives the following look.

javaCopy

@PWA(name="Project Base for Vaadin",shortName="Project Base")@Push@Theme(value="my-theme",variant=Lumo.DARK)publicclassAppShellimplementsAppShellConfigurator{}

Let’s go back to the method”configurePage ” in Vaadin Flow. Here, you can configure the general HTML shell of your application. This method customises various aspects of the HTML document, especially the “< head>"-Area. Here are the main adjustments you can make:

You can set the HTML page title. The page title is displayed in the browser tab and is essential for user experience and search engine optimisation.

Meta tags are essential for providing metadata about HTML documents. With “configurePage ” you can add different meta tags, such as:

• Viewport meta tag : This is especially important for responsive designs as it determines how the page will appear on different devices and screen sizes.
• Description : A meta tag to describe the content of your page, which is vital for search engine optimisation (SEO).
• Keywords : Keywords that describe the page’s content and can improve SEO.

Favicons are tiny icons that appear in browser tabs and bookmarks. You can set different sizes and types of favicons to ensure they look good on all devices and platforms. You can add links to external resources such as CSS stylesheets or JavaScript files. This is useful for including additional libraries or styles in your application.

For Progressive Web Apps (PWAs), you can use the “configurePage ” method to configure the manifest and service worker. The manifest defines how the application displays on the home screen of a mobile device while using the service worker for offline functionality and push notifications.

To increase the security of your application, you can add various security-related meta tags, such as Content Security Policy (CSP).

Social media tags such as Open Graph (for Facebook) or Twitter Cards can be added to determine how your pages appear when shared on social networks. For mobile applications, you can set specific settings that improve the appearance and behaviour of your web application on mobile devices, such as the status bar colour and the app icon on the home screen.

By using the method “configurePage ”, you can ensure that all critical meta information and resources are consistently and correctly integrated into every HTML page in your application. This contributes to a better user experience, optimised performance and improved web application security.

Which Are there any components?

After discussing the existing classes, we can modify the GUI a bit. We will now try out various components a little. The first question is, what categories of components are there in Vaadin Flow?

There are a variety of components in Vaadin Flow that can be grouped into different categories. Here are the main types of components with some examples:

Basic layout components:

• VerticalLayout : Arranges components vertically.
• HorizontalLayout : Arranges components horizontally.
• Div : A simple container that works like a<div> element in HTML.
• FormLayout : Explicitly designed for form layouts to arrange input fields in a grid.

Input components:

• TextField: Simple text input.
• TextArea: Multi-line text input.
• PasswordField: Input field for passwords in which the characters are hidden.
• ComboBox: A drop-down menu with choices.
• DatePicker: Selecting a date.
• CheckBox: A checkbox.
• RadioButtonGroup: A group of radio buttons.
• Select: A drop-down selection component.

Button components:

• Button : A simple button that can be clicked.
• NativeButton : A button that behaves like a native HTML button.
• Checkbox : A checkbox.

Display and information components:

• Label : A simple text label.
• Html : Component for displaying HTML content.
• Image : To view images.
• Notification : To view notifications.
• ProgressBar : A progress bar.
• Badge : To display small blocks of information or counters.

Navigation components:

• MenuBar : A menu bar.
• Tabs : Tabs for navigation between different views.
• Accordion : An accordion component for collapsible content.
• DrawerToggle : A toggle for a side menu.

Grid and data displays:

• Grid : A robust table for displaying data.
• TreeGrid : A tree-structured grid for displaying hierarchical data.
• ListBox : A simple list to display choices.
• DataView : To display data in a structured form.

Dialogues and overlays:

• Dialog : A modal dialogue for user interactions.
• Overlay : To overlay content on the current view.

Multimedia components:

• Audio : For embedding and playing audio files.
• Video : For embedding and playing video files.

Forms and validation:

• FormLayout : Used to arrange form fields in a layout.
• Binder : For binding forms to data models and validation.

Special and advanced components:

• Upload : To upload files.
• RichTextEditor : A WYSIWYG editor for rich text input.
• SplitLayout : To divide the layout into two areas using a sliding divider.
• DateTimePicker : To select the date and time.

These components cover many use cases and enable developers to create rich and interactive web applications. Vaadin Flow also offers the ability to create and extend custom components to meet specific needs.

Hello World for advanced users

Let’s now move on to the practical application of individual components. We are writing a small application that includes multiple input fields, a table to display data, and easy navigation between two views. This extension showcases more of Vaadin’s capabilities, including layouts, forms, data bindings, and navigation. However, we have not yet gone into the concepts of industrial applications. We will address this in detail in the next post.

If you have different views, the question almost automatically arises as to how you can define the general appearance of the application. Layouts in which the primary division of the application is defined are suitable for this. Vaadin Flow offers you the componentAppLayout at this point. This is a simple layout structure that already provides you with elements such as a navigation bar and a workspace. This is precisely what we will use now. For this, we create the classMainLayout and direct them fromAppLayout away. In the constructor, we define the desired structure. In our example, we would like to receive two entries for navigation. Firstly, access to the existing MainView and secondly to a new view with the nameDataView to record and display a personal data record. The classRouterLink is the required element to be added to a navigation bar.

javaCopy

publicclassMainLayoutextendsAppLayout{ publicMainLayout(){   createHeader(); } privatevoidcreateHeader(){   RouterLinkmainLink=newRouterLink("Hello World",MainView.class);   RouterLinkdataLink=newRouterLink("Data View",DataView.class);   addToNavbar(newHorizontalLayout(mainLink,dataLink)); }}

So the classMainView Now knowing that it is within the layout is in the annotation@Route the classMainLayout specified in the layout attribute.

javaCopy

@Route(value="",layout=MainLayout.class)publicclassMainViewextendsVerticalLayout{    //COLLAR}

This way, a nested layout can be quickly built with Vaadin Flow.

Now, let’s move on to the second view,DataView. Here, we can do the same thing asMainView proceeds.

xmlCopy

@Route(value = "data", layout = MainLayout.class)public class DataView extends VerticalLayout { private List<Person> people = new ArrayList<>(); private Grid<Person> grid = new Grid<>(Person.class); private ListDataProvider<Person> dataProvider     = new ListDataProvider<>(people); public DataView() {   was firstNameField = new TextField("First Name");   was lastNameField = new TextField("Last Name");   was addButton = new Button("Add Person",       event -> {         people.add(             new Person(firstNameField.getValue(),                        lastNameField.getValue()));         dataProvider.refreshAll();       });   grid.setDataProvider(dataProvider);   grid.setColumns("firstName", "lastName");   add(firstNameField, lastNameField, addButton, grid); }}

TheDataView is also derived from oneVerticalLayout. First, we define the required infrastructure elements. In this case, there is a list for the people, a table to display them, and aListDataProvider , which is the connection between the graphical representation and the data itself. I won’t go into the specific properties here, just this much in advance: the attributes of the classPerson are the headings of the columns in the table.

Now, we define the appearance and dynamic behaviour of the displayed components. This time, there are two text fields for entering the data and a button for adding the data to the table. TheActionListener of the button creates an instance of the class from the values ​​of the two input fieldsPerson. This instance is added to the list and subsequently informed to theDataProvider that there has been a change to the data. So that theDataProvider and the table know about each other, they are made known to each other. This is done by passing theDataProvider to the table.

Finally, all visible elements are added to theDataView.

If you start the application now, you will get the following views:

We now have a rudimentary basis for our projects with Vaadin Flow. The following article will look at how we can expand the project.

I have placed the example on Github using the following URL.

Until then, have fun with your experiments.

Happy Coding

Sven

]]>JavaVaadinhttps://svenruppert.com/posts/comparing-code-coverage-techniques-line-property-based-and-mutation-testing/Fri, 31 May 2024 09:20:30 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/comparing-code-coverage-techniques-line-property-based-and-mutation-testing/What is Test Coverage? Test coverage is a metric used in software testing to measure the testing performed on a piece of software. It indicates how thoroughly a software program has been tested by identifying which parts of the code have been executed (covered) during testing and which have not. Here are the key aspects of test coverage:<![CDATA[

What is Test Coverage?

Test coverage is a metric used in software testing to measure the testing performed on a piece of software. It indicates how thoroughly a software program has been tested by identifying which parts of the code have been executed (covered) during testing and which have not. Here are the key aspects of test coverage:

1. What is Test Coverage?
   1. Purpose:
   2. Examples of Types of Test Coverage:
   3. Benefits:
   4. Limitations:
2. What is the history of Test Coverage?
   1. Early Days of Software Testing (1950s-1960s)
   2. Introduction of Code Coverage (1970s)
   3. Development of Coverage Tools (1980s-1990s)
   4. Rise of Agile and Test-Driven Development (2000s)
   5. Modern Coverage Tools and Practices (2010s-Present)
   6. Key Figures and Influential Works
3. Trends and Future Directions
4. The Basic: Line Coverage
   1. Key Concepts of Line Coverage
   2. How Line Coverage Works
   3. Example Workflow with JaCoCo
   4. Interpreting Line Coverage Reports
   5. Best Practices
   6. Limitations
5. Property Based Testin Coverage
   1. Key Concepts of Property-Based Testing
   2. Popular Libraries for Property-Based Testing in Java
   3. How to Use Property-Based Testing in Java
   4. Benefits of Property-Based Testing
   5. Challenges and Best Practices
6. Mutation-Based Testing Coverage
   1. Key Concepts of Mutation Testing
   2. Steps in Mutation Testing
   3. Pitest for Mutation Testing in Java
   4. Example of Mutation Testing
   5. Benefits of Mutation Testing
   6. Challenges and Best Practices
7. Conclusion

Purpose :

The primary purpose of test coverage is to ensure that the code has been exercised to the maximum extent possible, reducing the chances of undiscovered bugs. It helps identify untested parts of the code, enabling testers to create additional tests to cover those areas.

Examples of Types of Test Coverage :

Code Coverage : Measures the extent to which the source code is tested. It includes various levels like:

Statement Coverage : Ensures that each statement in the code has been executed at least once.

Branch Coverage : Ensures that each branch (true/false) of control structures (like if statements) has been executed.

Function Coverage : Ensures that each function in the code has been called and executed.

Condition Coverage : Ensures that each boolean expression has been evaluated to be both true and false.

Feature Coverage : Measures how many features or requirements have been tested.

Path Coverage : Ensures that all possible paths in the code have been executed.

Benefits:

Improved Quality : Higher test coverage generally leads to higher quality software as more defects are likely to be found and fixed.

Risk Management : Identifying untested parts of the code helps manage risks by thoroughly testing critical areas.

Maintenance : Helps maintain the code by testing new changes and not introducing new bugs.

Limitations:

False Sense of Security : High coverage does not guarantee the software is bug-free. It only indicates that the tests have executed the code.

Focus on Quantity : Focusing too much on coverage percentages can lead to writing tests that simply increase coverage without effectively testing the functionality.

Complexity : Achieving 100% coverage can be difficult and time-consuming, especially for large and complex codebases.

Overall, test coverage is a valuable metric in the software development lifecycle. It provides insights into the effectiveness of the testing process and highlights areas that may need additional attention.

What is the history of Test Coverage?

The history of test coverage can be traced back to the evolution of software engineering and the increasing complexity of software systems, which necessitated more rigorous and systematic testing methodologies. Here are some key milestones and developments in the history of test coverage:

Early Days of Software Testing (1950s-1960s)

Initial Testing Practices : In the early days of computing, software testing was often ad hoc, with no formal methods or metrics. Testing was primarily focused on ensuring that the software ran without crashing.

Emergence of Formal Methods : The need for systematic testing approaches became evident as software systems became more complex. Early researchers and practitioners started formalising testing techniques.

Introduction of Code Coverage (1970s)

Code Coverage Concept : Code coverage emerged in the 1970s to measure how thoroughly software tests exercised the code. This period saw the development of various coverage criteria, such as statement coverage, branch coverage, and path coverage.

Myers’ Work : Glenford Myers’ book The Art of Software Testing, first published in 1979, was influential in promoting structured and systematic testing practices, including the use of coverage metrics.

Development of Coverage Tools (1980s-1990s)

Early Tools : In the 1980s and 1990s, early test coverage tools were developed that automated code coverage measurement. These tools helped developers and testers visualise and quantify the extent of their testing efforts.

Integration with Development Environments : As integrated development environments (IDEs) and automated testing frameworks became more common, coverage tools began to be integrated into these environments, making it easier for developers to incorporate coverage measurement into their workflows.

Rise of Agile and Test-Driven Development (2000s)

Agile Methodologies : The adoption of Agile methodologies in the early 2000s emphasised iterative development and continuous testing, leading to a greater focus on test coverage to ensure code quality.

Test-Driven Development (TDD) : TDD, a practice where tests are written before the code itself, gained popularity. This approach inherently promoted high levels of test coverage, as each piece of code was written to pass specific tests.

Modern Coverage Tools and Practices (2010s-Present)

Advanced Coverage Tools : Modern coverage tools have become more sophisticated, offering features such as real-time coverage analysis, integration with CI/CD pipelines, and detailed reporting. Examples include JaCoCo for Java, Istanbul for JavaScript, and Coverage.py for Python.

Shift-Left Testing : The shift-left testing approach, which advocates for testing early and often in the development lifecycle, has further reinforced the importance of test coverage. Tools and practices have evolved to support this approach, ensuring that coverage metrics are available throughout development.

Code Quality Platforms : Comprehensive code quality platforms like SonarQube have integrated test coverage metrics with other quality indicators, providing a holistic view of software health.

Key Figures and Influential Works

Glenford Myers : His seminal work “The Art of Software Testing” laid the foundation for systematic software testing and introduced many critical concepts related to test coverage.

Tom DeMarco and Tim Lister : Their work on software metrics and quality assurance highlighted the importance of measuring and improving software processes, including testing practices.

Trends and Future Directions

Increased Automation : As software development continues towards greater automation, test coverage tools are becoming more integrated with automated testing frameworks and CI/CD pipelines.

AI and Machine Learning : Emerging technologies like AI and machine learning are being applied to testing and coverage analysis, helping to identify untested areas more intelligently and predict potential risks based on coverage data.

Focus on Test Effectiveness : While achieving high test coverage is important, there is a growing emphasis on test effectiveness. This includes ensuring that tests not only cover code but also validate that the code behaves correctly under various conditions.

Test coverage has evolved significantly over the decades from a conceptual metric to a critical component of modern software development practices, helping to ensure the quality and reliability of software systems.

The Basic: Line Coverage

Line coverage is a code coverage metric that measures whether each line of source code in a program has been executed during testing. In Java, line coverage helps developers understand how thoroughly their tests exercise their code by indicating which lines have been tested and which have not. Here’s an in-depth look at line coverage in Java:

Key Concepts of Line Coverage

Definition : Statement coverage measures whether each line of code has been executed at least once during testing.

Importance :

• Bug Detection : Helps identify untested parts of the code that might contain bugs.
• Code Quality : Ensures that all parts of the code are tested, leading to better overall code quality.
• Maintenance : Aids in maintaining the code by ensuring that changes and new additions are tested.

Tools for Measuring Line Coverage in Java :

• JaCoCo (Java Code Coverage) : A widely used open-source library for measuring code coverage in Java projects.
• Emma : Another popular tool but has mainly been superseded by JaCoCo.
• Cobertura : An older code coverage tool that is still used in some projects.
• EclEmma : An Eclipse plugin for JaCoCo, making visualising coverage directly in the IDE easy.

How Line Coverage Works

Instrumentation : Tools like JaCoCo instrument the bytecode of a Java program to insert probes at various points. These probes collect data on which lines of code are executed during the test run.

Running Tests : Once the code is instrumented, the tests are executed. As the tests run, the probes collect execution data for each line of code.

Reporting : After the tests are complete, the coverage tool generates a report showing which lines of code were executed. This report often includes visual indicators (such as colour coding) to show covered (executed) and uncovered (not executed) lines.

Example Workflow with JaCoCo

Adding JaCoCo to the Project :

For a Maven project, you add the JaCoCo plugin to thepom.xml file:

xmlCopy

<build>       <plugins>         <plugin>           <groupId>org.jacoco</groupId>           <artifactId>jacoco-maven-plugin</artifactId>           <version>0.8.8</version>           <executions>             <execution>               <goals>                 <goal>prepare-agent</goal>               </goals>             </execution>             <execution>               <id>report</id>               <phase>test</phase>               <goals>                 <goal>report</goal>               </goals>             </execution>           </executions>         </plugin>       </plugins>     </build>

Running Tests :

Execute your tests using Maven:mvn clean test

Generating Coverage Report :

After running the tests, generate the coverage report:mvn jacoco:report

Viewing the Report :

The report is usually generated in the**target/site/jacoco** directory. To view the detailed coverage report, open theindex.html file in a web browser.

Interpreting Line Coverage Reports

Coverage Percentage : Indicates the proportion of lines executed versus the total number of lines.

Highlighted Lines : Typically, green lines indicate covered lines, while red lines indicate uncovered lines.

Detailed Metrics : Reports might include additional metrics like the number of covered and missed lines, branches, and methods.

Best Practices

Aim for High Coverage : While 100% coverage is ideal, strive for the highest coverage practical for your project to ensure maximum reliability.

Focus on Critical Code : Ensure that the most critical and complex parts of the code are covered extensively.

Regular Monitoring : Run coverage reports regularly as part of your CI/CD pipeline to catch regressions and ensure new code is adequately tested.

Complement with Other Testing Techniques : Line coverage is one aspect of testing. For comprehensive coverage, use it alongside other techniques like unit, integration, and manual testing.

Limitations

False Sense of Security : High-line coverage does not guarantee the absence of bugs. It’s possible to have tests covering all lines but not effectively testing the logic.

Focus on Quantity over Quality : Simply aiming for high coverage numbers can lead to writing superficial tests that do not profoundly validate the code’s behaviour.

Line coverage is a valuable metric for assessing the thoroughness of your tests in Java projects. Using tools like JaCoCo and following best practices ensures that your code is well-tested and maintains high-quality standards. However, it should be used as part of a broader testing strategy to achieve the best results.

Property Based Testin Coverage

Property-based testing (PBT) is a testing methodology where instead of writing individual test cases with specific inputs and expected outputs, you define properties that should hold true for a wide range of inputs. The testing framework then automatically generates test cases to verify these properties. This can uncover edge cases and unexpected behaviours that might not be evident with traditional example-based testing.

Key Concepts of Property-Based Testing

Properties : These are general assertions about the behaviour of your code. Properties describe the expected behaviour for various inputs rather than specific examples.

Generators : These automatically create input data for tests, allowing the exploration of a wide range of possible values.

Shrinkers : When a test fails, shrinkers reduce the size of the failing input to the simplest form that still causes the failure, making it easier to diagnose the problem.

Popular Libraries for Property-Based Testing in Java

jqwik : A modern property-based testing library for Java that integrates well with JUnit 5.

QuickTheories : A property-based testing tool inspired by QuickCheck from Haskell, focusing on creating and shrinking data for tests.

JUnit-QuickCheck : An extension for JUnit that brings property-based testing capabilities inspired by Haskell’s QuickCheck to Java.

How to Use Property-Based Testing in Java

Adding jqwik to Your Project :

For a Maven project, add the following dependency to yourpom.xml:

xmlCopy

<dependency>       <groupId>net.jqwik</groupId>       <artifactId>jqwik</artifactId>       <version>1.5.1</version>       <scope>test</scope>     </dependency>

Writing a Property-Based Test :

Define properties using the**@Property** annotation and use jqwik’s built-in generators to create input data.

javaCopy

importnet.jqwik.api.*;     publicclassExamplePropertyTest{         @Property         booleanconcatenationLength(@ForAllStringa,@ForAllStringb){             return(a+b).length()==a.length()+b.length();         }     }

Running the Tests :

Execute the tests using your preferred method, such as running through an IDE that supports JUnit 5 or using Maven:mvn test

Interpreting Results :

The framework automatically generates a wide range of inputs to test the property. If a property fails, jqwik attempts to shrink the failing input to a minimal case.

Benefits of Property-Based Testing

Increased Coverage : Automatically generates a large number of test cases, covering more scenarios and edge cases.

Uncovering Edge Cases : Helps find edge cases that might not be considered during example-based testing.

Less Manual Work : Reduces the need to manually write extensive individual test cases.

Better Specifications : Encourages writing more general and robust specifications of program behaviour.

Challenges and Best Practices

Defining Good Properties : One of the main challenges is defining general and meaningful properties. These properties should capture the code’s essential invariants and behaviours.

Handling Complex Input Data : It may be necessary to write custom generators and shrinkers for complex input data.

Balancing Between Unit and Property-Based Tests : Property-based testing is robust but should complement rather than replace traditional unit tests. Unit tests are still useful for specific scenarios and regression testing.

Property-based testing in Java provides a powerful tool for verifying that code behaves correctly across a wide range of inputs. Defineting properties and using automated input generation helps uncover bugs and edge cases that might be missed by traditional testing methods. While there are challenges in defining good properties and handling complex inputs, the benefits of increased coverage and robustness make it a valuable addition to any testing strategy.

Mutation-Based Testing Coverage

Mutation testing is a technique used to evaluate the quality of software tests. It involves modifying a program’s source code in small ways, called “mutants,” and then running the test suite to see if the tests detect these changes. A good test suite should be able to detect and fail when these mutants are introduced, indicating that the tests are effective in catching errors.

Key Concepts of Mutation Testing

Mutants : Variations of the original program created by making small changes, such as altering operators, changing constants, or modifying control flow.

Mutation Operators : Specific rules or patterns used to create mutants. Examples include:

Arithmetic Operator Replacement (AOR) : Replaces arithmetic operators like+ with-.

Logical Operator Replacement (LOR) : This function replaces logical operators like&& with||.

Relational Operator Replacement (ROR) : Replaces relational operators like> with<.

Mutation Score : The percentage of mutants that are detected (killed) by the test suite.

Steps in Mutation Testing

Generate Mutants : Apply mutation operators to the original source code to create a set of mutant programs.

Run Tests : Execute the test suite on each mutant.

Analyse Results : Determine if the tests pass or fail for each mutant. If a test fails, the mutant is considered “killed.” If all tests pass, the mutant is considered “survived.”

Pitest for Mutation Testing in Java

A widely used mutation testing tool for Java that integrates with various build tools and CI/CD pipelines.

Adding PIT to Your Project

Add the PIT plugin to yourpom.xml:

xmlCopy

<plugin>       <groupId>org.pitest</groupId>       <artifactId>pitest-maven</artifactId>       <version>1.6.9</version>       <executions>         <execution>           <goals>             <goal>mutationCoverage</goal>           </goals>         </execution>       </executions>     </plugin>

Running Mutation Tests

Run the mutation tests using:mvn org.pitest:pitest-maven:mutationCoverage

Interpreting Results

PIT generates a detailed HTML report showing the mutation score, which mutants were killed, and which survived. The report helps identify weak spots in your test suite.

Example of Mutation Testing

Consider a simple Java class and its test:

Calculator.java :

javaCopy

publicclassCalculator{    publicintadd(inta,intb){        returna+b;    }}

CalculatorTest.java :

javaCopy

import staticorg.junit.jupiter.api.Assertions.assertEquals;importorg.junit.jupiter.api.Test;publicclassCalculatorTest{    @Test    publicvoidtestAdd(){        Calculatorcalc=newCalculator();        assertEquals(5,calc.add(2,3));    }}

When running mutation testing on this example, a typical mutant might change the addition operator to subtraction:

javaCopy

publicintadd(inta,intb){    returna-b;}

If the test suite is adequate, it will fail for this mutant, thus killing it. The test suite might need improvement to cover this scenario if it doesn’t.

Benefits of Mutation Testing

Improves Test Quality : Helps identify weaknesses in the test suite and areas where tests might be missing.

Comprehensive Coverage : Provides a more rigorous evaluation of test effectiveness compared to code coverage metrics alone.

Identifies Redundant Tests : Helps detect tests that do not contribute to detecting faults in the code.

Challenges and Best Practices

Performance Overhead : Mutation testing can be time-consuming, especially for large codebases, as it requires running the test suite multiple times.

Equivalent Mutants : Some mutants may be logically equivalent to the original code and cannot be killed by any test, which can be challenging to identify and handle.

Selective Mutation Testing : To reduce the performance overhead, focus on critical parts of the codebase or use a subset of mutation operators.

Mutation testing is a powerful technique for assessing the effectiveness of your test suite by introducing small changes (mutants) to the code and checking if the tests detect these changes. Tools like PIT make it practical to integrate mutation testing into Java projects, providing valuable insights into test quality and helping to improve the robustness of your tests. By complementing traditional testing methods with mutation testing, you can achieve higher confidence in the correctness and reliability of your software.

Conclusion

Each coverage type has its strengths and weaknesses, making them suitable for different aspects of software testing:

Line Coverage is useful for ensuring that all parts of the code are executed, but should be complemented with other methods to ensure test quality.

Property-Based Coverage excels at testing the code’s behaviour over a wide range of inputs and can uncover edge cases that traditional testing might miss.

Mutation-Based Coverage provides a rigorous assessment of test suite effectiveness by ensuring that tests can detect intentional faults, though it is resource-intensive.

For a comprehensive testing strategy, it is beneficial to combine these approaches, leveraging the strengths of each to ensure both thorough and practical testing.

]]>JavaMutation TestingTDDhttps://svenruppert.com/posts/securing-apache-maven-understanding-cache-related-risks/Mon, 27 May 2024 14:30:22 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/securing-apache-maven-understanding-cache-related-risks/What is a Package Manager - Bird-Eye View A package manager is a tool or system in software development designed to simplify the process of installing, updating, configuring, and removing software packages on a computer system. It automates managing dependencies and resolving conflicts between different software components, making it easier for developers to work with various libraries, frameworks, and tools within their projects.<![CDATA[

What is a Package Manager - Bird-Eye View

A package manager is a tool or system in software development designed to simplify the process of installing, updating, configuring, and removing software packages on a computer system. It automates managing dependencies and resolving conflicts between different software components, making it easier for developers to work with various libraries, frameworks, and tools within their projects.

Package managers typically provide a centralised repository or repositories where software packages are hosted. Users can then use the package manager to search for, download, and install the desired packages and any necessary dependencies directly from these repositories.

1. What is a Package Manager - Bird-Eye View
2. What are the pros and cons of Package Managers?
   1. Pros:
      1. Simplified Installation:
      2. Dependency Management:
      3. Version Control:
      4. Centralised Repository:
      5. Consistency:
   2. Cons:
      1. Limited Control:
      2. Security Risks:
      3. Versioning Issues:
      4. Performance Overhead:
      5. Dependency Bloat:
3. What is Maven?
   1. Project Object Model (POM):
   2. Dependency Management:
   3. Convention over Configuration:
   4. Build Lifecycle:
   5. Plugins:
   6. Central Repository:
   7. Integration with IDEs:
   8. Transparency and Repeatability:
4. What are typical Securit Risks using Maven?
   1. Dependency vulnerabilities:
   2. Malicious dependencies:
   3. Repository compromise:
   4. Man-in-the-middle attacks:
   5. Build script injection:
5. How does the dependency-resolving mechanism work in Maven?
   1. Dependency Declaration:
   2. Dependency Tree:
   3. Repository Resolution:
   4. Dependency Download:
   5. Version Conflict Resolution:
   6. Transitive Dependency Resolution:
   7. Dependency Caching:
   8. Dependency Scope:
6. What attacks target Maven’s cache structure?
   1. Cache Poisoning:
   2. Cache Exfiltration:
   3. Cache Manipulation:
7. Conclusion:

Some popular package managers include:

1.APT (Advanced Package Tool) : Used primarily in Debian-based Linux distributions such as Ubuntu, APT simplifies installing and managing software packages.

2.YUM (Yellowdog Updater Modified) andDNF (Dandified YUM) : Package managers commonly used in Red Hat-based Linux distributions like Fedora and CentOS.

3.Homebrew : A package manager for macOS and Linux, Homebrew simplifies the installation of software packages and libraries.

4.npm (Node Package Manager) andYarn are package managers for JavaScript and Node.js that manage dependencies in web development projects.

5.pip : The package installer for Python, allowing developers to easily install Python libraries and packages from the Python Package Index (PyPI) repository.

6.Composer: is a dependency manager for PHP, used to manage libraries and dependencies in PHP projects.

Package managers greatly simplify software development by automating the process of installing and managing software dependencies. This reduces errors, improves efficiency, and facilitates collaboration among developers.

What are the pros and cons of Package Managers?

Package managers offer numerous benefits to software developers and users alike, but they also have drawbacks. Here are the pros and cons of using a package manager:

Pros:

Simplified Installation :

Package managers automate the process of installing software packages, making it straightforward for users to add new software to their systems.

Dependency Management:

Package managers handle dependencies automatically, ensuring all required libraries and components are installed correctly. This reduces the likelihood of dependency conflicts and makes it easier to manage complex software ecosystems.

Version Control:

Package managers keep track of software versions and updates, allowing users to upgrade to newer versions when they become available quickly. This helps ensure that software remains up-to-date and secure.

Centralised Repository:

Package managers typically provide access to a centralised repository of software packages, making it easy for users to discover new software and libraries.

Consistency:

Package managers enforce consistency across different environments by ensuring that all installations are performed in a standardised manner. This reduces the likelihood of configuration errors and compatibility issues.

Cons:

Limited Control:

Package managers abstract away many of the details of software installation and configuration, which can sometimes limit users’ control over the installation process. Advanced users may prefer more manual control over their software installations.

Security Risks:

Because package managers rely on centralised repositories, malicious actors could compromise them and distribute malicious software packages. Users must trust the integrity of the repository and the software packages hosted within it.

Versioning Issues:

Dependency management can sometimes lead to versioning issues, primarily when multiple software packages depend on conflicting versions of the same library. Resolving these conflicts can be challenging and may require manual intervention.

Performance Overhead:

Package managers introduce a performance overhead, as they must download and install software packages and their dependencies. In some cases, this overhead may be negligible, but it can become a concern for large projects or systems with strict performance requirements.

Dependency Bloat:

Dependency management can sometimes lead to “dependency bloat,” where software projects rely on a large number of external libraries and components. This can increase the size of software installations and introduce additional maintenance overhead.

While package managers offer significant benefits in simplifying software installation and dependency management, users must be aware of the potential drawbacks and trade-offs involved.

What is Maven?

Apache Maven is a powerful build automation and dependency management tool primarily used for Java projects. However, it can also manage projects in other languages like C#, Ruby, and Scala. It provides a consistent and standardised way to build, test, and deploy software projects. Here are some critical details about Maven:

Project Object Model (POM):

Maven uses a Project Object Model (POM) to describe a project’s structure and configuration. The POM is an XML file that contains information about the project’s dependencies, build settings, plugins, and other metadata. It serves as the central configuration file for the project and is used by Maven to automate various tasks.

Dependency Management:

One of Maven’s key features is its robust dependency management system. Dependencies are specified in the POM file, and Maven automatically downloads the required libraries from remote repositories such as Maven Central. Maven also handles transitive dependencies, automatically resolving and downloading dependencies required by other dependencies.

Convention over Configuration:

Maven follows the “convention over configuration” principle, which encourages standard project structures and naming conventions. By adhering to these conventions, Maven can automatically infer settings and reduce the configuration required.

Build Lifecycle:

Maven defines a standard build lifecycle consisting of phases: compile, test, package, install, and deploy. Each phase represents a specific stage in the build process, and Maven plugins can be bound to these phases to perform various tasks such as compiling source code, running tests, creating JAR or WAR files, and deploying artefacts.

Plugins:

Maven is highly extensible through plugins, which provide additional functionality for tasks such as compiling code, generating documentation, and deploying artefacts. Maven plugins can be either built-in or custom-developed, and they can be configured in the POM file to customise the build process.

Central Repository:

Maven Central is the default repository for hosting Java libraries and artefacts. It contains a vast collection of open-source and third-party libraries that can be easily referenced in Maven projects. Additionally, organisations and individuals can set up their own repositories to host proprietary or custom libraries.

Integration with IDEs:

Maven integrates seamlessly with popular Integrated Development Environments (IDEs) such as Eclipse, IntelliJ IDEA, and NetBeans. IDEs typically provide Maven support through plugins, allowing developers to import Maven projects, manage dependencies, and run Maven goals directly from the IDE.

Transparency and Repeatability:

Maven uses declarative configuration and standardised project structures to promote transparency and repeatability in the build process. This makes it easier for developers to understand and reproduce builds across different environments.

Overall, Apache Maven is a versatile and widely used tool in the Java ecosystem. It offers powerful features for automating build processes, managing dependencies, and promoting best practices in software development. Its adoption by numerous open-source projects and enterprises underscores its importance in modern software development workflows.

What are typical Securit Risks using Maven?

While Apache Maven is a widely used and generally secure tool for managing dependencies and building Java projects, there are some potential security risks associated with its usage:

Dependency vulnerabilities:

One of the leading security risks with Maven (as with any dependency management system) is the possibility of including dependencies with known vulnerabilities. If a project relies on a library with a security flaw, it could expose the application to various security risks, including remote code execution, data breaches, and denial-of-service attacks. It’s essential to update dependencies to patched versions regularly and use tools like OWASP Dependency-Check to identify and mitigate vulnerabilities.

Malicious dependencies:

While Maven Central and other reputable repositories have strict guidelines for publishing artefacts, there is still a risk of including malicious or compromised dependencies in a project. Attackers could compromise a legitimate library or create a fake library with malicious code and upload it to a repository. Developers should only use trusted repositories, verify the integrity of dependencies, and review code changes carefully.

Repository compromise:

Maven relies on remote repositories to download dependencies, and if a repository is compromised, it could serve malicious or tampered artefacts to unsuspecting users. While Maven Central has robust security measures, smaller or custom repositories may have weaker security controls. Organisations should implement secure repository management practices, such as using HTTPS for communication, signing artefacts with GPG, and restricting access to trusted users.

Man-in-the-middle attacks:

Maven communicates with remote repositories over HTTP or HTTPS, and if an attacker intercepts the communication, they could tamper with the downloaded artefacts or inject malicious code into the project. To mitigate this risk, developers should use HTTPS for all repository communications, verify SSL certificates, and consider using tools like Maven’s own repository manager or Nexus Repository Manager, which support repository proxies and caching to reduce the reliance on external repositories.

Build script injection:

Maven’s build scripts (POM files) are XML files that define project configurations, dependencies, and build settings. If an attacker gains unauthorised access to a project’s source code repository or CI/CD pipeline, they could modify the build script to execute arbitrary commands or introduce vulnerabilities into the build process. Organisations should implement proper access controls, code review processes, and security scanning tools to detect and prevent unauthorised changes to build scripts.

By understanding these security risks and implementing best practices for secure software development and dependency management, developers can mitigate potential threats and ensure the integrity and security of their Maven-based projects. Regular security audits, vulnerability scanning, and staying informed about security updates and patches are also essential for maintaining a secure development environment.

How does the dependency-resolving mechanism work in Maven?

Maven’s dependency resolution mechanism is a crucial aspect of its functionality, ensuring that projects have access to the necessary libraries and components while managing conflicts and versioning issues effectively. Here’s how the dependency-resolving mechanism works in Maven:

Dependency Declaration:

Developers specify dependencies for their projects in the project’s POM (Project Object Model) file. Dependencies are declared within the<dependencies> element, where each dependency includes details such as group ID, artifact ID, and version.

Dependency Tree:

Maven constructs a dependency tree based on the declared dependencies in the POM file. This tree represents the hierarchical structure of dependencies, including direct dependencies (specified in the POM file) and transitive dependencies (dependencies required by other dependencies).

Repository Resolution:

When a build is initiated, Maven attempts to resolve dependencies by searching for them in configured repositories. By default, Maven Central is the primary repository, but developers can specify additional repositories in the POM file or through Maven settings.

Dependency Download:

If a dependency is not already present in the local Maven repository, Maven downloads it from the remote repository where it’s hosted. Maven stores downloaded dependencies in the local repository (~/.m2/repository by default) for future reuse.

Version Conflict Resolution:

Maven employs a strategy for resolving version conflicts when multiple dependencies require different versions of the same library. By default, Maven uses the “nearest-wins” strategy, where the version closest to the project in the dependency tree takes precedence. Developers can explicitly specify versions for dependencies to override the default resolution behaviour. Maven also provides options such as dependency management and exclusions to manage version conflicts better.

Transitive Dependency Resolution:

Maven automatically resolves transitive dependencies, ensuring all required libraries and components are included in the project’s classpath. Maven traverses the dependency tree recursively, downloading and including transitive dependencies as needed.

Dependency Caching:

Maven caches downloaded dependencies in the local repository to improve build performance and reduce network traffic. Subsequent builds reuse cached dependencies whenever possible, avoiding redundant downloads.

Dependency Scope:

Maven supports different dependency scopes (e.g., compile, test, runtime) to control the visibility and usage of dependencies during various phases of the build lifecycle. Scopes help prevent unnecessary dependencies in production builds and improve build efficiency.

Overall, Maven’s dependency resolution mechanism simplifies the management of project dependencies by automating the process of downloading, organizing, and resolving dependencies. This allows developers to focus on writing code rather than manually managing library dependencies.

What attacks target Maven’s cache structure?

Attacks targeting the cache structure of Maven, particularly the local repository cache (~/.m2/repository), are relatively rare but can potentially exploit vulnerabilities in the caching mechanism to compromise the integrity or security of Maven-based projects. Here are some potential attacks that could target Maven’s cache structure:

Cache Poisoning:

Description : Cache poisoning attacks involve manipulating the contents of the local repository cache to introduce malicious artefacts or modified versions of legitimate artefacts. Once poisoned, subsequent builds may unwittingly use the compromised artefacts, leading to security vulnerabilities or system compromise.

Attack Vector : Attackers may exploit vulnerabilities in Maven’s caching mechanism, such as improper input validation or insecure handling of cached artefacts, to inject malicious artefacts into the cache.

Mitigation : To mitigate cache poisoning attacks, Maven users should regularly verify the integrity of cached artefacts using checksums or digital signatures. Employing secure repository management practices, such as signing artefacts with GPG and enabling repository managers with artefact validation, can also enhance security.

Cache Exfiltration:

Description : Cache exfiltration attacks involve an attacker extracting sensitive information from the local repository cache. This could include credentials, private keys, or other confidential data inadvertently stored within cached artefacts or metadata.

Attack Vector : Attackers may exploit vulnerabilities in Maven or its plugins to access and extract sensitive information stored in the local repository cache. For example, a compromised plugin could inadvertently leak credentials stored in Maven settings files.

Mitigation : To mitigate cache exfiltration attacks, developers should avoid storing sensitive information in Maven configuration files or cached artefacts. Instead, they should use secure credential management practices, such as environment variables or encrypted credential stores, to prevent the inadvertent exposure of sensitive data.

Cache Manipulation:

Description : Cache manipulation attacks involve modifying the contents of the local repository cache to alter the behaviour of Maven builds or compromise the integrity of projects. To achieve their objectives, attackers may tamper with cached artefacts, metadata, or configuration files.

Attack Vector : Attackers may exploit vulnerabilities in Maven or its dependencies to manipulate the local repository cache. For example, an attacker could modify cached artefacts to include backdoors or malware.

Mitigation : To mitigate cache manipulation attacks, developers should ensure that the local repository cache is stored in a secure location with restricted access permissions. They should also regularly verify the integrity of cached artifacts and configuration files using checksums or digital signatures. Implementing secure software development practices, such as code reviews and vulnerability scanning, can also help detect and prevent cache manipulation attacks.

While attacks targeting Maven’s cache structure are relatively uncommon, developers should remain vigilant and implement security best practices to safeguard against potential vulnerabilities and threats. Regularly updating Maven and its dependencies to the latest versions and maintaining awareness of security advisories and patches is essential for mitigating the risk of cache-related attacks.

Conclusion:

In conclusion, while attacks explicitly targeting Maven’s cache structure are relatively rare, they pose potential risks to the integrity and security of Maven-based projects. Cache poisoning, cache exfiltration, and cache manipulation are possible attack vectors that attackers may exploit to compromise Maven’s local repository cache (~/.m2/repository).

To mitigate these risks, developers and organisations should adopt robust security measures and best practices:

Regularly Verify Artifact Integrity : Employ mechanisms such as checksums or digital signatures to verify the integrity of cached artefacts and ensure they haven’t been tampered with.

Secure Credential Management : Avoid storing sensitive information, such as credentials or private keys, in Maven configuration files or cached artefacts. Instead, use secure credential management practices, such as environment variables or encrypted stores.

Access Control and Permissions : Ensure the local repository cache is stored securely with restricted access permissions to prevent unauthorised access or manipulation.

Update and Patch : Regularly update Maven and its dependencies to the latest versions to mitigate potential vulnerabilities. Stay informed about security advisories and apply patches promptly.

Secure Repository Management : Implement secure repository management practices, including signing artefacts with GPG, enabling repository managers with artefact validation, and using HTTPS for repository communication.

By implementing these security measures and remaining vigilant, developers can reduce the likelihood of cache-related attacks and enhance the overall security posture of Maven-based projects. Additionally, fostering a culture of security awareness and education within development teams can help mitigate risks and respond effectively to emerging threats in the software development lifecycle.

]]>JavaSecurityTDDhttps://svenruppert.com/posts/cwe-22-best-practices-to-use-java-nio/Wed, 22 May 2024 10:30:27 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cwe-22-best-practices-to-use-java-nio/In today’s digital landscape, ensuring the security of your applications is paramount. One critical vulnerability developers must guard against is CWE-22, Path Traversal. This vulnerability can allow attackers to access files and directories outside the intended scope, potentially leading to unauthorised access and data breaches.<![CDATA[

In today’s digital landscape, ensuring the security of your applications is paramount. One critical vulnerability developers must guard against is CWE-22, Path Traversal. This vulnerability can allow attackers to access files and directories outside the intended scope, potentially leading to unauthorised access and data breaches.

Java’s New I/O (NIO) package provides robust tools for file and path manipulation, which can be effectively leveraged to mitigate the risks associated with CWE-22. This blog post will explore best practices for using Java NIO to prevent path traversal vulnerabilities. From proper path normalisation to secure file handling techniques, we’ll cover everything you need to know to enhance the security of your Java applications.

1. Normalise Paths
2. Validate Path to Ensure It Stays Within a Base Directory
3. Use Secure Directory and File Permissions
   1. Setting Permissions for a Directory
   2. Setting Permissions for Files
   3. Checking File Permissions
   4. Practical Security Considerations
4. Handle Symlinks Securely
   1. Example: Avoid Following Symlinks
   2. Example: Validate the Target of Symlink
   3. Practical Security Considerations
5. Validate User Inputs Rigorously
   1. Example Implementation
6. Implement Logging and Monitoring
   1. Detecting Suspicious Activity
   2. Incident Response and Forensics
   3. Accountability and Compliance
   4. Proactive Threat Detection
   5. Improving Security Posture
   6. Real-Time Monitoring
7. Use SecureRandom for Temporary Files
8. Use FileSystems for Consistent Path Handling
   1. Key Concepts
   2. Best Practices
   3. Example Implementation

Normalise Paths

Normalisation eliminates any redundant elements from a path, such as. (current directory) and.. (parent directory). This process helps ensure that paths are appropriately structured and prevents path traversal attacks.

Create the Base Path :

Define the base directory against which the user input will be resolved. This should be the directory within which you want to restrict access.

javaCopy

**PathbasePath=Paths.get("/var/www/uploads").normalize();**

Resolve the User Input :

Combine the base path with the user-provided input to create a complete path. This step is crucial to ensure that any relative paths the user provides are interpreted in the context of the base path.

javaCopy

StringuserInput=request.getParameter("file");PathresolvedPath=basePath.resolve(userInput);

Normalise the Resolved Path :

Normalise the resolved path to eliminate any redundant elements. This step ensures that any. or.. in the path are correctly resolved.

javaCopy

**PathnormalizedPath=resolvedPath.normalize();**

Validate the Normalized Path :

Ensure that the normalised path starts with the base path. This check ensures that the path does not traverse outside the intended directory.

javaCopy

if(!normalizedPath.startsWith(basePath)){thrownewSecurityException("Invalid file path: path traversal attempt detected.");}

Validate Path to Ensure It Stays Within a Base Directory

Normalise the Base Path :

Ensure that the base directory is normalised to its simplest form.

Resolve and Normalize the User-Provided Path :

Combine the base directory with the user-provided path, then normalise the resultant path.

Check if the Normalized Path Starts with the Base Path :

Ensure the final normalised path starts with the base directory to confirm it hasn’t traversed outside the intended directory.

Use Secure Directory and File Permissions

Using secure directory and file permissions is essential to ensuring your files’ and directories’ safety and integrity, especially when dealing with user-provided inputs in Java applications.

Java NIO provides APIs for setting and checking file permissions. ThePosixFilePermissions andPosixFileAttributeView classes can do this.

Setting Permissions for a Directory

To set permissions for a directory, you can useFiles.createDirectories andPosixFilePermissions to specify the desired permissions:

xmlCopy

import java.nio.file.*;import java.nio.file.attribute.PosixFilePermission;import java.nio.file.attribute.PosixFilePermissions;import java.util.Set;import java.io.IOException;public class SecureFileHandler { /** * Creates a directory with secure permissions. * * @param dirPath The path of the directory to create. * @throws IOException if an I/O error occurs. */ public static void createSecureDirectory(Path dirPath) throws IOException { Set<PosixFilePermission> perms = PosixFilePermissions.fromString("rwxr-x---"); FileAttribute<Set<PosixFilePermission>> attr = PosixFilePermissions.asFileAttribute(perms); Files.createDirectories(dirPath, attr); } /** * Example usage of creating a secure directory. * * @param args Command line arguments. */ public static void main(String[] args) { Path dirPath = Paths.get("/var/www/uploads"); try { createSecureDirectory(dirPath); } catch (IOException e) { e.printStackTrace(); } }}

Setting Permissions for Files

Similarly, you can set permissions for files using theFiles.setPosixFilePermissions method:

xmlCopy

import java.nio.file.*;import java.nio.file.attribute.PosixFilePermission;import java.nio.file.attribute.PosixFilePermissions;import java.io.IOException;import java.util.Set;public class SecureFileHandler { /** * Sets secure permissions for a file. * * @param filePath The path of the file. * @throws IOException if an I/O error occurs. */ public static void setSecureFilePermissions(Path filePath) throws IOException { Set<PosixFilePermission> perms = PosixFilePermissions.fromString("rw-r-----"); Files.setPosixFilePermissions(filePath, perms); } /** * Example usage of setting secure file permissions. * * @param args Command line arguments. */ public static void main(String[] args) { Path filePath = Paths.get("/var/www/uploads/example.txt"); try { setSecureFilePermissions(filePath); } catch (IOException e) { e.printStackTrace(); } }}

Checking File Permissions

Before performing file operations, checking if the file or directory has the correct permissions is important. Here’s how you can do this:

xmlCopy

import java.nio.file.*;import java.nio.file.attribute.PosixFilePermissions;import java.nio.file.attribute.PosixFilePermission;import java.io.IOException;import java.util.Set;public class SecureFileHandler { /** * Checks if a file has the required permissions. * * @param filePath The path of the file. * @param requiredPerms The required permissions. * @return True if the file has the required permissions, false otherwise. * @throws IOException if an I/O error occurs. */ public static boolean hasRequiredPermissions(Path filePath, Set<PosixFilePermission> requiredPerms) throws IOException { Set<PosixFilePermission> perms = Files.getPosixFilePermissions(filePath); return perms.containsAll(requiredPerms); } /** * Example usage of checking file permissions. * * @param args Command line arguments. */ public static void main(String[] args) { Path filePath = Paths.get("/var/www/uploads/example.txt"); Set<PosixFilePermission> requiredPerms = PosixFilePermissions.fromString("rw-r-----"); try { if (hasRequiredPermissions(filePath, requiredPerms)) { System.out.println("File has the required permissions."); } else { System.out.println("File does not have the required permissions."); } } catch (IOException e) { e.printStackTrace(); } }}

Practical Security Considerations

Restrict Write Access :

Ensure that write access is limited to necessary users and processes only. This minimises the risk of unauthorised modifications.

Read-Only Access :

Set read-only permissions for files that do not need to be modified to prevent unauthorised changes.

Execute Permissions :

Be cautious when granting executing permissions. Only grant execute permissions to necessary users and ensure scripts or executables are secure.

Owner and Group Permissions :

Set appropriate owner and group permissions. Ensure sensitive files and directories are owned by the correct user and group.

Symbolic Links :

Avoid following symbolic links if possible. This can prevent attackers from bypassing security controls using symbolic link attacks.

Use of umask :

Configure theumask value to control the default permission settings for newly created files and directories. This ensures a baseline of security.

Using Java NIO’sPath andFiles classes and proper permission settings can significantly enhance the security of your file and directory operations. These practices help mitigate the risk of unauthorised access and modifications, protecting your applications from potential vulnerabilities related to CWE-22 (Path Traversal).

Handle Symlinks Securely

Handling symbolic links (symlinks) securely is crucial to prevent potential security risks such as bypassing access controls and path traversal attacks. Symlinks can be exploited by attackers to gain unauthorised access to files and directories. Here are best practices for securely handling symlinks in Java using the NIO (New I/O) API:

Avoid Following Symlinks :

Use theNOFOLLOW_LINKS option when performing file operations to avoid following symlinks. This ensures operations are performed on the symlink rather than the target file or directory.

Validate the Target of Symlinks :

If your application must follow symlinks, validate the symlink’s target to ensure it points to an allowed location.

Check for Symlinks :

Explicitly check if a path is a symlink and handle it accordingly.

Example: Avoid Following Symlinks

Here’s how you can avoid following symlinks in file operations using Java NIO:

javaCopy

importjava.nio.file.*;importjava.nio.file.attribute.BasicFileAttributes;importjava.io.IOException;publicclassSecureSymlinkHandler{/** * Checks if the given path is a symbolic link. * * @param path The path to check. * @return True if the path is a symbolic link, false otherwise. * @throws IOException if an I/O error occurs. */publicstaticbooleanisSymlink(Pathpath)throwsIOException{returnFiles.isSymbolicLink(path);}/** * Safely deletes a file without following symbolic links. * * @param path The path to the file to delete. * @throws IOException if an I/O error occurs. */publicstaticvoidsafeDelete(Pathpath)throwsIOException{if(isSymlink(path)){thrownewSecurityException("Refusing to delete symbolic link: "+path);}Files.delete(path);}/** * Safely reads a file's attributes without following symbolic links. * * @param path The path to the file. * @return The file's attributes. * @throws IOException if an I/O error occurs. */publicstaticBasicFileAttributessafeReadAttributes(Pathpath)throwsIOException{returnFiles.readAttributes(path,BasicFileAttributes.class,LinkOption.NOFOLLOW_LINKS);}publicstaticvoidmain(String[]args){Pathpath=Paths.get("/var/www/uploads/example.txt");try{if(isSymlink(path)){System.out.println("Path is a symbolic link.");}else{System.out.println("Path is not a symbolic link.");BasicFileAttributesattrs=safeReadAttributes(path);System.out.println("File size: "+attrs.size());safeDelete(path);System.out.println("File deleted safely.");}}catch(IOException|SecurityExceptione){e.printStackTrace();}}}

Example: Validate the Target of Symlink

If your application needs to follow symlinks, validate their targets to ensure they point to an acceptable location

javaCopy

importjava.nio.file.*;importjava.io.IOException;publicclassSecureSymlinkHandler{/** * Validates that the symlink's target is within the allowed base directory. * * @param symlink The symbolic link to validate. * @param baseDir The allowed base directory. * @throws IOException if an I/O error occurs or if validation fails. */publicstaticvoidvalidateSymlinkTarget(Pathsymlink,PathbaseDir)throwsIOException{if(!Files.isSymbolicLink(symlink)){thrownewIllegalArgumentException("Path is not a symbolic link: "+symlink);}Pathtarget=Files.readSymbolicLink(symlink).normalize();PathresolvedTarget=baseDir.resolve(target).normalize();if(!resolvedTarget.startsWith(baseDir)){thrownewSecurityException("Invalid symlink target: "+resolvedTarget);}}publicstaticvoidmain(String[]args){Pathsymlink=Paths.get("/var/www/uploads/symlink");PathbaseDir=Paths.get("/var/www/uploads").normalize();try{validateSymlinkTarget(symlink,baseDir);System.out.println("Symlink target is valid and within the allowed base directory.");}catch(IOException|SecurityExceptione){e.printStackTrace();}}}

Practical Security Considerations

Restrict Symlink Creation :

Only allow trusted users to create symlinks. This minimises the risk of symlinks being used for malicious purposes.

Regularly Audit Symlinks :

Periodically audit symlinks within your application to ensure they are not pointing to unauthorised locations.

Use Symlink Aware Libraries :

Use libraries that are aware of and handle symlinks securely. This can help mitigate the risk of unintentional symlink following.

Least Privilege Principle :

Ensure that your application runs with the minimum required privileges. This reduces the impact of potential symlink-related vulnerabilities.

Deploy Defense in Depth :

Use multiple layers of security controls to protect against symlink attacks. These include filesystem permissions, application-level checks, and regular monitoring.

Validate User Inputs Rigorously

Validating user inputs rigorously is crucial to ensure the security and integrity of an application. Proper input validation helps prevent various attacks, including path traversal (CWE-22), SQL injection, cross-site scripting (XSS), and more. Here are some best practices and techniques to rigorously validate user inputs in Java applications:

Whitelist Validation : Only allow inputs that match a predefined set of acceptable values. This is the most secure form of validation.

Blacklist Validation : Reject inputs that contain known dangerous characters or patterns. This approach is less secure than whitelisting but can be used as an additional measure.

Length Checks : Ensure that inputs are within the expected length limits. This prevents buffer overflows and denial of service (DoS) attacks.

Data Type Checks : Verify that inputs match the expected data type (e.g., integers, dates).

Encoding and Escaping : Encode and escape inputs to prevent injection attacks in different contexts (e.g., HTML, SQL).

Canonicalisation : Convert inputs to a standard format before validation. This helps compare and process inputs securely.

Restrict Allowed Characters :

Validate file names and paths against a whitelist of allowed characters. Reject any input containing characters that can alter the path structure (e.g.,..,/,\\).

Check Path Against Base Directory :

Ensure the resolved and normalised path starts with the intended base directory.

Example Implementation

Below is an example implementation in Java that combines these principles and techniques to validate user inputs, specifically for file paths:

xmlCopy

import java.nio.file.*;import java.util.Set;import java.util.HashSet;import java.util.logging.Logger;import java.io.IOException;import java.util.regex.Pattern;public class SecureInputValidator { private static final Logger logger = Logger.getLogger(SecureInputValidator.class.getName()); private static final Set<String> ALLOWED_EXTENSIONS = new HashSet<>(); static { ALLOWED_EXTENSIONS.add(".txt"); ALLOWED_EXTENSIONS.add(".jpg"); ALLOWED_EXTENSIONS.add(".png"); ALLOWED_EXTENSIONS.add(".pdf"); } /** * Validates the user-provided file name against a whitelist of allowed characters and extensions. * * @param fileName The user-provided file name. * @throws IllegalArgumentException if the file name is invalid. */ public static void validateFileName(String fileName) throws IllegalArgumentException { if (fileName == null || fileName.isEmpty()) { throw new IllegalArgumentException("File name cannot be null or empty."); } // Check for invalid characters Pattern pattern = Pattern.compile("[^a-zA-Z0-9._-]"); if (pattern.matcher(fileName).find()) { throw new IllegalArgumentException("File name contains invalid characters."); } // Check for allowed file extensions boolean validExtension = ALLOWED_EXTENSIONS.stream().anyMatch(fileName::endsWith); if (!validExtension) { throw new IllegalArgumentException("File extension is not allowed."); } } /** * Validates the user-provided path to ensure it stays within the base directory. * * @param baseDir The base directory. * @param userInput The user-provided input. * @return The validated and normalized path. * @throws SecurityException if a path traversal attempt is detected. * @throws IllegalArgumentException if the file name is invalid. * @throws IOException if an I/O error occurs. */ public static Path getSecureFilePath(String baseDir, String userInput) throws SecurityException, IllegalArgumentException, IOException { validateFileName(userInput); // Normalize the base directory Path basePath = Paths.get(baseDir).normalize(); // Resolve the user input against the base directory and normalize the result Path resolvedPath = basePath.resolve(userInput).normalize(); // Validate that the resolved path starts with the base directory if (!resolvedPath.startsWith(basePath)) { logSuspiciousActivity(userInput); throw new SecurityException("Invalid file path: path traversal attempt detected."); } return resolvedPath; } /** * Logs suspicious activity for further analysis. * * @param userInput The suspicious user input. */ private static void logSuspiciousActivity(String userInput) { logger.warning("Suspicious file access attempt: " + userInput); } /** * Example usage of the secure file path validation. * * @param args Command line arguments. */ public static void main(String[] args) { String baseDir = "/var/www/uploads"; String userInput = "example.txt"; try { Path filePath = getSecureFilePath(baseDir, userInput); System.out.println("Validated file path: " + filePath); } catch (SecurityException | IllegalArgumentException | IOException e) { e.printStackTrace(); } }}

Implement Logging and Monitoring

Logging and monitoring are essential to effectively prevent or deal with CWE-22 (Path Traversal) vulnerabilities. Here’s why these practices are critical:

Detecting Suspicious Activity

Logging and monitoring allow you to detect suspicious activities that might indicate a path traversal attempt. By capturing and analysing logs, you can identify unusual patterns or attempts to access files and directories that should be off-limits.

Example : Logging all file access attempts, including the requested paths, can help you spot anomalies such as attempts to use../ sequences to access parent directories.

javaCopy

privatestaticfinalLoggerlogger=Logger.getLogger(SecureFileHandler.class.getName());publicstaticvoidlogFileAccessAttempt(StringfilePath){logger.info("File access attempt: "+filePath);}

Incident Response and Forensics

In a security incident, having detailed logs can help in forensic analysis to understand how the attack was carried out. This information is vital for closing security gaps and preventing future attacks.

Example : Detailed logs can show the sequence of operations leading up to a detected breach, including the exact user inputs and system responses.

javaCopy

try{PathfilePath=resolveFilePath(userInput);logFileAccessAttempt(filePath.toString());}catch(SecurityExceptione){logger.warning("Security exception: "+e.getMessage());throwe;}

Accountability and Compliance

Many regulatory frameworks and standards require logging and monitoring as part of their compliance requirements. Implementing these practices ensures that you meet legal and regulatory obligations.

Example : Compliance with standards such as PCI-DSS or GDPR often requires comprehensive logging of security-related events to ensure accountability.

Proactive Threat Detection

Continuous monitoring allows you to detect and respond to threats proactively before they cause significant damage. Real-time monitoring solutions can alert you to potential path traversal attacks as they occur.

Example : Implementing real-time alerts for suspicious file access patterns can help take immediate corrective actions.

javaCopy

// Example of setting up a real-time alert system (pseudo-code)if(detectedPathTraversalAttempt){alertSecurityTeam("Potential path traversal attack detected: "+attemptedPath);}

Improving Security Posture

Regularly analysing logs and monitoring data helps improve overall security posture by identifying weaknesses and refining security policies.

Example : Reviewing access logs can highlight frequent user errors or misconfigurations that could be exploited, allowing you to address these proactively.

Real-Time Monitoring

Use monitoring tools and services to keep track of log data and generate alerts for suspicious activities. Tools like ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, or cloud-based solutions such as AWS CloudWatch and Azure Monitor can be useful.

javaCopy

// Pseudo-code for setting up real-time monitoring and alertsif(detectPathTraversal(logFileAccessAttempt)){triggerAlert("Potential path traversal attack detected: "+logFileAccessAttempt);}

Implementing logging and monitoring is not just a best practice but a necessary component of a robust security strategy. It helps in the early detection of threats, compliance with regulatory standards, effective incident response, and overall security posture improvement. Ensuring that all file access attempts and related activities are logged and monitored can better protect your applications from CWE-22 and other security vulnerabilities.

Use SecureRandom for Temporary Files

UsingSecureRandom to create temporary files in Java ensures better security and randomness, reducing the risk of vulnerabilities such as CWE-22. Here are best practices for usingSecureRandom for this purpose:

Instantiate SecureRandom : Create an instance ofSecureRandom to generate random values for temporary file names or other purposes requiring secure randomness.

javaCopy

**SecureRandomsecureRandom=newSecureRandom();**

Generate Secure Random Values : UseSecureRandom to generate a sequence of bytes that can be converted into a string or used directly in file names.

javaCopy

byte[]randomBytes=newbyte[16];secureRandom.nextBytes(randomBytes);StringrandomString=newBigInteger(1,randomBytes).toString(16);

Create Temporary Files with Secure Names : Create temporary files using the random string generated bySecureRandom. This helps prevent predictable file names, reducing the risk of collisions or attacks.

javaCopy

PathtempDir=Paths.get(System.getProperty("java.io.tmpdir"));PathtempFile=tempDir.resolve("tempFile_"+randomString+".tmp");Files.createFile(tempFile);

Ensure File Permissions : Set appropriate file permissions to prevent unauthorized users from accessing the temporary files.

javaCopy

**Files.setPosixFilePermissions(tempFile,PosixFilePermissions.fromString("rw-------"));**

Handle SecureRandom Properly : Ensure thatSecureRandom is not reseeded unnecessarily, as it can reduce the randomness quality. Initialise it once and reuse the instance.

javaCopy

SecureRandomsecureRandom=newSecureRandom();//UsesecureRandomthroughouttheapplication

Usejava.nio.file for File Operations : Utilize the NIO package to benefit from its security features and avoid common pitfalls associated with older IO methods.

javaCopy

**PathtempFile=Files.createTempFile(tempDir,"tempFile_",".tmp");**

Use FileSystems for Consistent Path Handling

UsingFileSystems in Java for consistent path handling ensures that paths are managed in a compatible way across different platforms and file systems. Here are some best practices and techniques for leveragingFileSystems in Java:

Key Concepts

FileSystems Class :

Thejava.nio.file.FileSystems class provides factory methods for creating file system instances and obtaining default file systems.

Path Interface :

Thejava.nio.file.Path interface represents a file or directory path in a platform-independent way.

Standardise Path Creation :

UseFileSystems to create paths consistently.

Best Practices

Obtain the Default FileSystem

The default file system corresponds to the platform’s file system on which the Java virtual machine runs.

javaCopy

importjava.nio.file.FileSystem;importjava.nio.file.FileSystems;FileSystemdefaultFileSystem=FileSystems.getDefault();

Create Paths Using the FileSystem

CreatePath objects using theFileSystem to ensure paths are created consistently.

javaCopy

importjava.nio.file.Path;Pathpath=defaultFileSystem.getPath("/var/www/uploads","example.txt");

Handle Paths Consistently Across Platforms

UsingFileSystems helps create paths compatible with different operating systems.

javaCopy

PathwindowsPath=defaultFileSystem.getPath("C:\\Users\\Public\\Documents","example.txt");PathunixPath=defaultFileSystem.getPath("/home/user/docs","example.txt");

Use Paths for Safe File Operations

UsingPaths fromFileSystems ensures that file operations are performed safely and consistently.

javaCopy

importjava.nio.file.Files;importjava.io.IOException;try{if(!Files.exists(path)){Files.createFile(path);}// Perform file operations}catch(IOExceptione){e.printStackTrace();}

Normalise and Resolve Paths

Always normalise and resolve paths to avoid relative and symlinks issues.

javaCopy

PathbasePath=defaultFileSystem.getPath("/var/www/uploads").normalize();PathuserPath=basePath.resolve("example.txt").normalize();if(!userPath.startsWith(basePath)){thrownewSecurityException("Path traversal attempt detected");}

Example Implementation

Here is an example demonstrating the use ofFileSystems for consistent path handling:

javaCopy

importjava.nio.file.FileSystem;importjava.nio.file.FileSystems;importjava.nio.file.Path;importjava.nio.file.Files;importjava.nio.file.attribute.PosixFilePermissions;importjava.io.IOException;publicclassSecurePathHandler{privatestaticfinalFileSystemfileSystem=FileSystems.getDefault();publicstaticPathcreateSecureTempFile(Stringprefix,Stringsuffix)throwsIOException{PathtempDir=fileSystem.getPath(System.getProperty("java.io.tmpdir"));PathtempFile=Files.createTempFile(tempDir,prefix,suffix);Files.setPosixFilePermissions(tempFile,PosixFilePermissions.fromString("rw-------"));returntempFile;}publicstaticPathresolveAndValidatePath(StringbaseDir,StringuserInput)throwsSecurityException,IOException{PathbasePath=fileSystem.getPath(baseDir).normalize();PathresolvedPath=basePath.resolve(userInput).normalize();if(!resolvedPath.startsWith(basePath)){thrownewSecurityException("Invalid file path: path traversal attempt detected.");}returnresolvedPath;}publicstaticvoidmain(String[]args){try{PathtempFile=createSecureTempFile("tempFile_",".tmp");System.out.println("Temporary file created: "+tempFile);StringbaseDir="/var/www/uploads";StringuserInput="example.txt";PathfilePath=resolveAndValidatePath(baseDir,userInput);System.out.println("Validated file path: "+filePath);}catch(IOException|SecurityExceptione){e.printStackTrace();}}}

UsingFileSystems for consistent path handling in Java ensures that your application can handle paths in a platform-independent manner. This helps in creating secure, reliable, and maintainable file-handling code. Following these best practices can prevent common pitfalls such as path traversal vulnerabilities and ensure that file operations are performed safely and consistently.

]]>JavaSecure Coding Practiceshttps://svenruppert.com/posts/cwe-22-improper-limitation-of-a-pathname-to-a-restricted-directory/Tue, 21 May 2024 14:33:53 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cwe-22-improper-limitation-of-a-pathname-to-a-restricted-directory/CWE-22, commonly called “Path Traversal,” is a vulnerability when an application fails to appropriately limit the paths users can access through a user-provided input. This can allow attackers to access directories and files outside the intended directory, leading to unauthorised access and potential system compromise. This vulnerability is particularly significant in Java applications due to the ubiquitous use of file handling and web resources. This document will delve into the nature of CWE-22, its implications, exploitation methods, and, most importantly, strategies to mitigate such vulnerabilities in Java applications.<![CDATA[

CWE-22, commonly called “Path Traversal,” is a vulnerability when an application fails to appropriately limit the paths users can access through a user-provided input. This can allow attackers to access directories and files outside the intended directory, leading to unauthorised access and potential system compromise. This vulnerability is particularly significant in Java applications due to the ubiquitous use of file handling and web resources. This document will delve into the nature of CWE-22, its implications, exploitation methods, and, most importantly, strategies to mitigate such vulnerabilities in Java applications.

1. Understanding CWE-22
2. Implications of CWE-22
3. Exploitation Techniques
4. Mitigation Strategies in Java
5. Case Study: A Vulnerable Java Application
   1. Vulnerable Code
   2. Secure Code
6. What Java Libraries are Helping against CWE-22
   1. Apache Commons IO
   2. OWASP Java Encoder
   3. Apache Shiro
   4. Spring Security
   5. Tika
   6. ESAPI (Enterprise Security API)
   7. Java NIO (New I/O)
   8. Hibernate Validator
7. What CVEs are based on CWE-22
   1. CVE-2020-9484
   2. CVE-2019-0232
   3. CVE-2018-11784

Understanding CWE-22

CWE-22 is categorised under “Improper Limitation of a Pathname to a Restricted Directory (‘Path Traversal’).” This occurs when an application constructs a path using user input without proper validation or sanitisation, allowing the user to specify paths that traverse outside the intended directory. Sequences such as../ can be used to move up the directory structure.

For example, consider the following Java code snippet that reads a file based on user input:

javaCopy

StringfileName=request.getParameter("file");Filefile=newFile("/var/www/uploads/"+fileName);FileInputStreamfis=newFileInputStream(file);

If thefileName parameter is not properly validated, an attacker can manipulate it to access files outside the/var/www/uploads/ directory, such as/etc/passwd, by using a path traversal sequence (../../etc/passwd).

Implications of CWE-22

The implications of CWE-22 can be severe, ranging from unauthorised access to sensitive files to full system compromise. Some of the potential impacts include:

Exposure of Sensitive Information : Attackers can access sensitive files such as configuration files, passwords, and personal data.

Data Integrity Compromise : Attackers can modify files, potentially leading to data corruption or alteration of critical application files.

Denial of Service (DoS) : Attackers can disrupt normal operations by accessing and modifying system files.

Execution of Arbitrary Code : In extreme cases, attackers might execute arbitrary code if they gain access to executable files or scripts.

Exploitation Techniques

To exploit a path traversal vulnerability, an attacker typically uses special characters and patterns in the input to navigate the file system. Here are some standard techniques:

Dot-Dot-Slash (../) : The most common method where the attacker uses../ to move up the directory structure.

Encoded Characters : Attackers may use URL encoding (%2e%2e%2f) or other encoding schemes to bypass simple input filters.

Null Byte Injection : Sometimes, null bytes (%00) are used to terminate strings early, effectively ignoring any appended extensions or path components.

Mitigation Strategies in Java

Mitigating CWE-22 in Java involves a combination of secure coding practices, input validation, and proper use of APIs. Here are some detailed strategies:

Canonicalisation and Normalization :

Ensure that the file paths are normalised before use. Java provides methods to normalise paths, which can help mitigate traversal attacks.

javaCopy

importjava.nio.file.Paths;importjava.nio.file.Path;publicFilegetFile(StringfileName)throwsIOException{PathbasePath=Paths.get("/var/www/uploads/");PathfilePath=basePath.resolve(fileName).normalize();if(!filePath.startsWith(basePath)){thrownewSecurityException("Attempted path traversal attack detected");}returnfilePath.toFile();}

Input Validation and Sanitization :

Perform strict validation on user inputs. Reject any input that contains potentially malicious patterns such as../, URL-encoded sequences, or other traversal patterns.

javaCopy

publicStringsanitizeFileName(StringfileName){if(fileName==null||fileName.contains("..")||fileName.contains("/")||fileName.contains("\\")){thrownewIllegalArgumentException("Invalid file name");}returnfileName;}

Whitelist Approach :

Use a whitelist approach to validate filenames against a set of allowed values or patterns. This can be more effective than blacklisting known destructive patterns.

javaCopy

publicbooleanisValidFileName(StringfileName){returnfileName.matches("[a-zA-Z0-9._-]+");}

File Access Controls :

Restrict file permissions on the server to limit access only to necessary files and directories. This reduces the impact of potential exploits.

Logging and Monitoring :

Implement comprehensive logging and monitoring to detect and respond to suspicious activities. Logs should capture sufficient details to trace potential exploitation attempts.

javaCopy

importjava.util.logging.Logger;publicclassFileAccessLogger{privatestaticfinalLoggerLOGGER=Logger.getLogger(FileAccessLogger.class.getName());publicvoidlogAccessAttempt(StringfileName){LOGGER.warning("Attempted access to file: "+fileName);}}

Case Study: A Vulnerable Java Application

Let’s consider a hypothetical case study to illustrate the application of these mitigation strategies. Suppose we have a simple Java web application that allows users to download files from the server.

Vulnerable Code

javaCopy

@WebServlet("/download")publicclassFileDownloadServletextendsHttpServlet{protectedvoiddoGet(HttpServletRequestrequest,HttpServletResponseresponse)throwsServletException,IOException{StringfileName=request.getParameter("file");Filefile=newFile("/var/www/uploads/"+fileName);if(file.exists()){FileInputStreamfis=newFileInputStream(file);response.setContentType("application/octet-stream");response.setHeader("Content-Disposition","attachment; filename=\""+file.getName()+"\"");byte[]buffer=newbyte[4096];intbytesRead;while((bytesRead=fis.read(buffer))!=-1){response.getOutputStream().write(buffer,0,bytesRead);}fis.close();}else{response.sendError(HttpServletResponse.SC_NOT_FOUND);}}}

This servlet reads thefile parameter from the request and constructs aFile object. Without proper validation, an attacker can exploit this to download arbitrary files from the server.

Secure Code

javaCopy

@WebServlet("/download")publicclassFileDownloadServletextendsHttpServlet{protectedvoiddoGet(HttpServletRequestrequest,HttpServletResponseresponse)throwsServletException,IOException{StringfileName=sanitizeFileName(request.getParameter("file"));PathbasePath=Paths.get("/var/www/uploads/");PathfilePath=basePath.resolve(fileName).normalize();if(!filePath.startsWith(basePath)){response.sendError(HttpServletResponse.SC_FORBIDDEN,"Invalid file path");return;}Filefile=filePath.toFile();if(file.exists()){FileInputStreamfis=newFileInputStream(file);response.setContentType("application/octet-stream");response.setHeader("Content-Disposition","attachment; filename=\""+file.getName()+"\"");byte[]buffer=newbyte[4096];intbytesRead;while((bytesRead=fis.read(buffer))!=-1){response.getOutputStream().write(buffer,0,bytesRead);}fis.close();}else{response.sendError(HttpServletResponse.SC_NOT_FOUND);}}privateStringsanitizeFileName(StringfileName){if(fileName==null||fileName.contains("..")||fileName.contains("/")||fileName.contains("\\")){thrownewIllegalArgumentException("Invalid file name");}returnfileName;}}

In the secure version, thesanitizeFileName method ensures the file name is free from traversal sequences, and the path is normalised and checked to prevent directory traversal.

CWE-22, or path traversal, is a critical vulnerability that can have severe consequences if not adequately mitigated. In Java applications, it is essential to employ a combination of secure coding practices, input validation, canonicalisation, and access controls to safeguard against this threat. By understanding the nature of CWE-22 and implementing robust security measures, developers can significantly reduce the risk of unauthorised file access and protect their applications from potential exploitation.

What Java Libraries are Helping against CWE-22

Several Java libraries and tools can help developers prevent CWE-22 (Path Traversal) vulnerabilities by providing robust input validation, path normalisation, and security mechanisms. Here are some of the most notable ones:

Apache Commons IO

Apache Commons IO provides a variety of utilities for working with the file system, which can help prevent path traversal vulnerabilities.

FilenameUtils : This class contains methods for normalising and validating file paths.

javaCopy

importorg.apache.commons.io.FilenameUtils;StringbasePath="/var/www/uploads/";StringfileName=FilenameUtils.normalize(request.getParameter("file"));if(!FilenameUtils.directoryContains(basePath,basePath+fileName)){thrownewSecurityException("Attempted path traversal attack detected");}

OWASP Java Encoder

The OWASP Java Encoder library encodes untrusted input to help protect against injection attacks, including those involving path traversal.

Encoder : Provides methods to encode user input for various contexts, including filenames, safely.

javaCopy

importorg.owasp.encoder.Encode;StringsafeFileName=Encode.forJava(request.getParameter("file"));

Apache Shiro

Apache Shiro is a powerful security framework that provides robust access control mechanisms for enforcing file access policies.

Path Permissions : Shiro allows you to define fine-grained access control policies related to file access.

javaCopy

// Define file access permissions in Shiro configuration[urls]/var/www/uploads/**=authc,perms["file:read"]

Spring Security

Spring Security is a comprehensive security framework that integrates seamlessly with Spring applications, providing mechanisms for authentication and authorisation.

Access Control : Spring Security can be configured to enforce strict access controls on file resources.

javaCopy

@PreAuthorize("hasPermission(#filePath, 'read')")publicvoidreadFile(PathfilePath){// read file logic}

Tika

Apache Tika is a library for detecting and extracting metadata and content from various file types. It includes utilities for working with file paths safely.

Tika IOUtils : Utility methods for safe file operations.

javaCopy

importorg.apache.tika.io.IOUtils;StringsafeFileName=IOUtils.toString(request.getParameter("file"),StandardCharsets.UTF_8);

ESAPI (Enterprise Security API)

The OWASP ESAPI library provides a comprehensive set of security controls, including file-handling utilities that can help prevent path traversal attacks.

Validator : Use ESAPI’s Validator to validate filenames.

javaCopy

importorg.owasp.esapi.ESAPI;importorg.owasp.esapi.Validator;Validatorvalidator=ESAPI.validator();StringsafeFileName=validator.getValidInput("file name",request.getParameter("file"),"Filename",255,false);

Java NIO (New I/O)

Java NIO provides modern APIs for file operations, including path manipulation and validation.

Path and Files : Usejava.nio.file.Path andjava.nio.file.Files for secure file operations.

javaCopy

importjava.nio.file.Path;importjava.nio.file.Paths;importjava.nio.file.Files;PathbasePath=Paths.get("/var/www/uploads/");PathfilePath=basePath.resolve(request.getParameter("file")).normalize();if(!filePath.startsWith(basePath)){thrownewSecurityException("Attempted path traversal attack detected");}if(Files.exists(filePath)){// read file logic}

Hibernate Validator

Hibernate Validator, the reference implementation of the Bean Validation API, can be used to enforce validation constraints on user inputs.

Custom Constraints : Define custom validation constraints for filenames.

javaCopy

importjavax.validation.constraints.Pattern;publicclassFileRequest{@Pattern(regexp="[a-zA-Z0-9._-]+")privateStringfileName;// getters and setters}

The Java libraries and tools provide robust mechanisms to prevent CWE-22 (Path Traversal) vulnerabilities. Using these libraries, developers can ensure that user inputs are appropriately validated, paths are normalised, and access controls are strictly enforced. This multi-layered approach significantly reduces the risk of unauthorised file access and enhances the overall security of Java applications.

What CVEs are based on CWE-22

Several Common Vulnerabilities and Exposures (CVEs) related to Java applications have been reported that are based on CWE-22, the Improper Limitation of a Pathname to a Restricted Directory (‘Path Traversal’). These CVEs highlight path traversal vulnerabilities’ real-world implications and impact in various Java-based systems and libraries. Here are some notable examples:

CVE-2020-9484

Description : Apache Tomcat HTTP/2 Request Smuggling and Path Traversal.

Affected Versions : Apache Tomcat 9.0.0.M1 to 9.0.35, 8.5.0 to 8.5.55, and 7.0.0 to 7.0.104.

Details : This vulnerability allowed an attacker to upload files to arbitrary locations via a specially crafted request. The issue was related to the improper handling of user input in file upload paths, leading to a path traversal vulnerability.

Mitigation : Upgrade to the latest versions of Apache Tomcat that include the patch for this vulnerability.

CVE-2019-0232

Description : Apache Tomcat Remote Code Execution via CGI Servlet.

Affected Versions : Apache Tomcat 9.0.0.M1 to 9.0.17, 8.5.0 to 8.5.39, and 7.0.0 to 7.0.93.

Details : This CVE was related to a path traversal vulnerability that allowed attackers to achieve remote code execution by manipulating the CGI Servlet configuration.

Mitigation : Disable the CGI Servlet if it is unnecessary or upgrade to a version that fixes the vulnerability.

CVE-2018-11784

Description : Apache JMeter Path Traversal Vulnerability.

Affected Versions : Apache JMeter 3.0 to 4.0.

Details : This vulnerability allowed attackers to access files outside the intended directories via a path traversal attack, leveraging improper file path validation in the application.

Mitigation : Upgrade to Apache JMeter 5.0 or later versions where the issue has been resolved.

These CVEs highlight the importance of addressing CWE-22 vulnerabilities in Java applications. Regularly updating libraries, frameworks, and application code is crucial to mitigating such vulnerabilities. Developers should adopt best practices for input validation, path normalisation, and robust security configurations to protect against path traversal attacks.

]]>JavaSecure Coding Practiceshttps://svenruppert.com/posts/cwe-416-use-after-free-vulnerabilities-in-java/Fri, 17 May 2024 12:17:30 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cwe-416-use-after-free-vulnerabilities-in-java/CWE-416: Use After Free Use After Free (UAF) is a vulnerability that occurs when a program continues to use a pointer after it has been freed. This can lead to undefined behaviour, including crashes, data corruption, and security vulnerabilities. The problem arises because the memory referenced by the pointer may be reallocated for other purposes, potentially allowing attackers to exploit the situation.<![CDATA[

CWE-416: Use After Free

Use After Free (UAF) is a vulnerability that occurs when a program continues to use a pointer after it has been freed. This can lead to undefined behaviour, including crashes, data corruption, and security vulnerabilities. The problem arises because the memory referenced by the pointer may be reallocated for other purposes, potentially allowing attackers to exploit the situation.

1. CWE-416: Use After Free
   1. Causes:
   2. Consequences:
2. CWE-416 in Java
   1. Scenario 1: Mismanagement of External Resources
      1. Automatic Resource Management with Try-With-Resources:
      2. Explicit Nullification:
   2. Scenario 2: Inappropriate Use of Weak References
      1. Strong References for Critical Data:
      2. Check References Before Use:
3. What CVE´s are based on CWE-416 in Java
   1. CVE-2022-45146:
   2. CVE-2023-38703:
4. What Design Patterns are used to avoid CWE-416 in Java?
   1. RAII (Resource Acquisition Is Initialization) Pattern
   2. Factory Pattern
   3. Singleton Pattern
   4. Observer Pattern
   5. Decorator Pattern
5. RAII (Resource Acquisition Is Initialization) Pattern in Java without using try-with-resources
   1. Example: RAII Implementation without try-with-resources
   2. Important Considerations:
6. Cleaner API Example:
7. More details about java.lang.ref.Cleaner, please
   1. How to Usejava.lang.ref.Cleaner:
   2. Example:
   3. Benefits:

Causes:

Incorrect memory management : Forgetting to nullify pointers after freeing memory.

Dangling pointers : Retaining references to freed memory and accessing it later.

Double freeing : Freeing memory more than once, causing subsequent operations on the exact memory location.

Consequences:

Security vulnerabilities : Attackers can exploit UAF to execute arbitrary code, escalate privileges, or cause denial of service.

Data corruption : UAF can lead to unexpected changes in data, causing program instability or incorrect behaviour.

Program crashes : Accessing freed memory can cause segmentation faults or other runtime errors.

CWE-416 in Java

With its automatic garbage collection, Java inherently protects against many types of memory management issues, including Use After Free (UAF) vulnerabilities. However, UAF-like issues can still occur in Java if resources other than memory (like file handles or network connections) are mismanaged. Below are some scenarios in Java that resemble UAF and strategies to mitigate them.

Scenario 1: Mismanagement of External Resources

In Java, even though memory is managed by the garbage collector, other resources like file handles or network sockets can be mismanaged, leading to use-after-close problems.

Example :

javaCopy

importjava.io.*;publicclassUseAfterFreeExample{publicstaticvoidmain(String[]args){FileInputStreamfis=null;try{fis=newFileInputStream("example.txt");// Use the file input stream...}catch(IOExceptione){e.printStackTrace();}finally{if(fis!=null){try{fis.close();}catch(IOExceptione){e.printStackTrace();}}}// Attempt to use the file input stream after it has been closedtry{intdata=fis.read();// This will throw an IOException}catch(IOExceptione){System.out.println("Attempted to read from a closed stream.");}}}

In this example, after closing theFileInputStream, attempting to read from it results in anIOException.

Automatic Resource Management with Try-With-Resources:

Use the try-with-resources statement introduced in Java 7, which ensures each resource is closed at the end of the statement.

javaCopy

importjava.io.*;publicclassUseAfterFreeFixed{publicstaticvoidmain(String[]args){try(FileInputStreamfis=newFileInputStream("example.txt")){// Use the file input stream...}catch(IOExceptione){e.printStackTrace();}// fis is automatically closed here// Attempting to use fis here will cause a compilation error}}

Explicit Nullification:

Set references tonull after closing them to avoid accidental reuse.

javaCopy

importjava.io.*;publicclassUseAfterFreeWithNull{publicstaticvoidmain(String[]args){FileInputStreamfis=null;try{fis=newFileInputStream("example.txt");// Use the file input stream...}catch(IOExceptione){e.printStackTrace();}finally{if(fis!=null){try{fis.close();}catch(IOExceptione){e.printStackTrace();}finally{fis=null;// Prevent reuse}}}// fis is now null, so the following attempt to use it will fail at compile-timeif(fis!=null){try{intdata=fis.read();// This will not compile since fis is null}catch(IOExceptione){System.out.println("Attempted to read from a closed stream.");}}}}

Scenario 2: Inappropriate Use of Weak References

While not a direct UAF issue, misusing weak references can lead to problems where objects are accessed after being reclaimed by the garbage collector.

xmlCopy

import java.lang.ref.WeakReference;public class WeakReferenceExample { public static void main(String[] args) { Object obj = new Object(); WeakReference<Object> weakRef = new WeakReference<>(obj); obj = null; // Eligible for garbage collection System.gc(); // Suggest garbage collection // Attempt to use the object after it may have been collected if (weakRef.get() != null) { System.out.println("Object is still alive."); } else { System.out.println("Object has been garbage collected."); } }}

In this example, after nullifying the strong reference and suggesting garbage collection, accessing the weak reference may result in null, indicating the object has been collected.

Strong References for Critical Data:

Avoid using weak references for critical objects that are still needed.

Check References Before Use:

Always check if a weak reference has been cleared before using it.

sqlCopy

if(weakRef.get()!=null){//SafetousetheobjectObjectobj=weakRef.get();//Useobj}else{//HandlethecasewheretheobjecthasbeencollectedSystem.out.println("Object has been garbage collected.");}

While Java’s automatic memory management reduces the risk of traditional UAF vulnerabilities, developers must still be cautious with resource management and weak references to avoid similar issues. Modern Java features like try-with-resources and being mindful of object references can help maintain robust and error-free code.

What CVE´s are based on CWE-416 in Java

Here are some notable CVEs related to CWE-416: Use After Free vulnerabilities in Java:

CVE-2022-45146:

This vulnerability is found in the FIPS Java API of Bouncy Castle BC-FJA before version 1.0.2.4. Changes in the JVM garbage collector in Java 13 and later versions can trigger this issue, causing temporary keys used by the module to be zeroed out while still in use. This results in errors or potential information loss. FIPS compliance users are unaffected as it applies to Java 7, 8, and 11.

CVE-2023-38703:

This CVE affects the PJSIP multimedia communication library, written in multiple languages, including Java. The issue arises when the synchronisation between higher-level SRTP media and lower-level transport, such as UDP, is not maintained, leading to a use-after-free vulnerability. This can result in unexpected application termination or memory corruption.

These examples highlight the importance of proper memory and resource management in Java applications, even though the language manages memory through garbage collection. For more information on these vulnerabilities, visit the National Vulnerability Database (NVD) or specific advisories on platforms like GitHub and mvnrepository.

What Design Patterns are used to avoid CWE-416 in Java?

Several design patterns and best practices can be employed to ensure robust and safe memory and resource management in Java to avoid CWE-416 (Use After Free). While Java’s garbage collector reduces the risk of traditional UAF vulnerabilities, the following patterns and practices help avoid related issues with resources like file handles, sockets, or other external resources.

RAII (Resource Acquisition Is Initialization) Pattern

RAII is a design pattern that ties resource management to object lifetime. Although RAII is more commonly associated with C++, Java can leverage a similar approach using try-with-resources.

javaCopy

try(FileInputStreamfis=newFileInputStream("example.txt")){// Use the file input stream...}catch(IOExceptione){e.printStackTrace();}//fisisautomaticallyclosedhere

The try-with-resources statement ensures that each resource is closed at the end of the statement, reducing the risk of resource leaks and use-after-close issues.

Factory Pattern

The Factory Pattern encapsulates an object’s creation logic, allowing for central management of resource creation and ensuring that resources are properly initialized and managed.

javaCopy

publicclassConnectionFactory{publicstaticConnectioncreateConnection(){// Implement resource creation and managementreturnnewConnection();}}// UsageConnectionconnection=ConnectionFactory.createConnection();

This pattern centralises resource management, making it easier to enforce consistent resource-handling practices.

Singleton Pattern

The Singleton Pattern ensures that a class has only one instance and provides a global point of access to it. This is particularly useful for managing shared resources like database connections or thread pools.

javaCopy

publicclassDatabaseConnection{privatestaticDatabaseConnectioninstance;privateDatabaseConnection(){// Initialize the connection}publicstaticsynchronizedDatabaseConnectiongetInstance(){if(instance==null){instance=newDatabaseConnection();}returninstance;}}

By controlling a single instance, the Singleton Pattern helps manage the lifecycle and closure of shared resources properly.

Observer Pattern

The Observer Pattern can notify interested parties about resource lifecycle events, such as the availability or closing of a resource. This ensures that all application parts know the resource state and can act accordingly.

xmlCopy

public interface ResourceListener { void onResourceClosed();}public class Resource { private List<ResourceListener> listeners = new ArrayList<>(); public void addListener(ResourceListener listener) { listeners.add(listener); } public void close() { // Close the resource notifyListeners(); } private void notifyListeners() { for (ResourceListener listener : listeners) { listener.onResourceClosed(); } }}

Observers are notified of resource state changes, allowing them to update their behaviour accordingly.

Decorator Pattern

The Decorator Pattern allows for dynamic functionality extensions, including resource management features like logging or additional cleanup actions when resources are closed.

javaCopy

publicclassResourceDecoratorimplementsAutoCloseable{privatefinalAutoCloseableresource;publicResourceDecorator(AutoCloseableresource){this.resource=resource;}@Overridepublicvoidclose()throwsException{// Additional cleanup actionsresource.close();}}// Usagetry(ResourceDecoratordecoratedResource=newResourceDecorator(newFileInputStream("example.txt"))){// Use the resource}

This pattern enhances the resource management behaviour without modifying the original resource classes.

Developers can use these patterns to ensure better resource management and mitigate the risk of use-after-free vulnerabilities in Java applications.

RAII (Resource Acquisition Is Initialization) Pattern in Java without using try-with-resources

RAII (Resource Acquisition Is Initialization) is a design pattern that binds the lifecycle of resources to the lifetime of objects, ensuring resources are adequately released when the object goes out of scope. While Java does not directly support RAII due to its garbage collection system and lack of deterministic destruction, you can still achieve similar behaviour without using try-with-resources by employing custom management classes and finalisers.

Here’s how you can implement RAII in Java using a custom resource management class and finalisers:

Example: RAII Implementation without try-with-resources

Create a Resource Management Class :

This class will manage the resource and ensure it is adequately released when the object is no longer needed.

javaCopy

publicclassManagedResource{privatebooleanisReleased=false;publicManagedResource(){// Acquire the resourceSystem.out.println("Resource acquired");}publicvoiduseResource(){if(isReleased){thrownewIllegalStateException("Resource has been released");}// Use the resourceSystem.out.println("Resource is being used");}publicvoidrelease(){if(!isReleased){// Release the resourceSystem.out.println("Resource released");isReleased=true;}}@Overrideprotectedvoidfinalize()throwsThrowable{try{release();}finally{super.finalize();}}}

Use the Resource Management Class :

Ensure the resource is managed correctly by explicitly calling the release method or relying on the finaliser.

javaCopy

publicclassRAIIExample{publicstaticvoidmain(String[]args){ManagedResourceresource=newManagedResource();try{resource.useResource();// Perform operations with the resource}finally{resource.release();}// Alternatively, without explicit release, the finalizer will handle itManagedResourceresource2=newManagedResource();resource2.useResource();// No explicit release call}}

Explanation :

Resource Management Class : TheManagedResource class acquires the resource during initialisation and provides auseResource method to use it. Therelease method releases the resource.

Finalizer : Thefinalize method ensures the resource is released when theManagedResource object is garbage collected ifrelease was not called explicitly.

Usage : In theRAIIExample class, the resource is explicitly released in thefinally block. For the second resource (resource2), if not explicitly released, the finalizer will handle it.

Important Considerations:

Finalizers : Using finalizers can be unpredictable, as the timing of garbage collection is not guaranteed. Due to performance implications and unpredictability, it is generally recommended to avoid relying on finalizers for critical resource management.

Explicit Resource Management : Whenever possible, explicitly manage resources using well-defined methods or blocks (like try-with-resources) to ensure timely and predictable resource release.

Java Cleaner API : For better control over resource management, consider using the Java Cleaner API introduced in Java 9. This API provides a more flexible and reliable alternative to finalizers.

Cleaner API Example:

javaCopy

importjava.lang.ref.Cleaner;publicclassManagedResource{privatestaticfinalCleanercleaner=Cleaner.create();privatefinalCleaner.Cleanablecleanable;privatebooleanisReleased=false;publicManagedResource(){this.cleanable=cleaner.register(this,this::release);System.out.println("Resource acquired");}publicvoiduseResource(){if(isReleased){thrownewIllegalStateException("Resource has been released");}System.out.println("Resource is being used");}publicvoidrelease(){if(!isReleased){System.out.println("Resource released");isReleased=true;}}@Overrideprotectedvoidfinalize()throwsThrowable{try{release();}finally{super.finalize();}}}

The Cleaner API provides a more controlled and predictable way to manage resources without relying on the finalisation mechanism. It ensures that resources are cleaned up promptly when the object becomes unreachable.

Using these techniques, you can effectively manage resources in Java, ensuring they are properly released and avoiding issues like CWE-416.

More details about java.lang.ref.Cleaner, please

Thejava.lang.ref.Cleaner class, introduced in Java 9, is a modern replacement for the deprecatedjava.lang.ref.Finalizer mechanism. It provides a more flexible and efficient way to clean up resources when an object becomes unreachable.

How to Usejava.lang.ref.Cleaner:

Create a Cleaner Instance :

sqlCopy

Cleanercleaner=Cleaner.create();

Define a Cleanable Resource :

A cleanable resource must implement aRunnable interface to define the cleanup action.

javaCopy

classStateimplementsRunnable{@Overridepublicvoidrun(){// Cleanup actionSystem.out.println("Resource cleaned up");}}

Register the Resource with Cleaner :

Register the resource and its cleanup action with theCleaner.

javaCopy

classManagedResource{privatefinalCleaner.Cleanablecleanable;publicManagedResource(Cleanercleaner){Statestate=newState();this.cleanable=cleaner.register(this,state);System.out.println("Resource acquired");}}

Using the Managed Resource :

javaCopy

publicclassCleanerExample{publicstaticvoidmain(String[]args){Cleanercleaner=Cleaner.create();ManagedResourceresource=newManagedResource(cleaner);// Use the resource...// No explicit cleanup is required, the Cleaner will handle it}}

Example:

Here’s a complete example demonstrating the usage ofjava.lang.ref.Cleaner:

javaCopy

importjava.lang.ref.Cleaner;publicclassCleanerExample{staticclassStateimplementsRunnable{@Overridepublicvoidrun(){// Cleanup actionSystem.out.println("Resource cleaned up");}}staticclassManagedResource{privatefinalCleaner.Cleanablecleanable;publicManagedResource(Cleanercleaner){Statestate=newState();this.cleanable=cleaner.register(this,state);System.out.println("Resource acquired");}}publicstaticvoidmain(String[]args){Cleanercleaner=Cleaner.create();ManagedResourceresource=newManagedResource(cleaner);// Use the resource...// No explicit cleanup required, the Cleaner will handle it}}

Benefits:

Predictability :Cleaner ensures that cleanup actions are executed promptly, unlike finalizers which have unpredictable execution.

Performance : UsingCleaner avoids the performance issues associated with finalizers, such as increased GC pressure and potential for memory leaks.

Simplicity : The API is straightforward, making it easier to implement and maintain.

By usingjava.lang.ref.Cleaner, Java developers can achieve efficient and predictable resource management, minimising risks associated with manual resource cleanup.

]]>JavaSecure Coding PracticesSecurityhttps://svenruppert.com/posts/basics-of-the-gauss-kruger-coordinate-system/Thu, 16 May 2024 14:34:19 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/basics-of-the-gauss-kruger-coordinate-system/Transverse Mercator Projection : The Gauss-Krüger system uses the transverse Mercator projection, which means the cylindrical projection is rotated 90 degrees. This allows for better accuracy over long north-south extents.<![CDATA[

Transverse Mercator Projection :

The Gauss-Krüger system uses the transverse Mercator projection, which means the cylindrical projection is rotated 90 degrees. This allows for better accuracy over long north-south extents.

1. Transverse Mercator Projection:
2. Ellipsoid:
3. Zones:
4. Coordinates:
5. Carl Friedrich Gauss (1777-1855)
6. Johann Heinrich Louis Krüger (1857-1923)
7. Development of the Gauss-Krüger Coordinate System
8. Technical Features
9. Modern Developments and Usage
10. Step-by-Step Conversion Process
11. Example Conversion
12. Online Tools:
13. Using Java:

Ellipsoid :

The system is based on an ellipsoid model of the Earth, which is more accurate than a spherical model. Different regions might use slightly different ellipsoids, but a common one used in Europe is the Bessel 1841 ellipsoid.

Zones :

The Gauss-Krüger system divides the area into longitudinal zones that are 3° wide. Each zone has its own central meridian. This helps to reduce distortions within each zone. Zone numbering usually starts at a prime meridian (often 9° E or 15° E) and increases by 3° for each zone.

Coordinates :

Coordinates are expressed in meters. The system uses false easting and false northing to ensure that all coordinates within a zone are positive.

Easting (X) : Measured in meters from the zone’s central meridian.

Northing (Y) : Measured in meters from the equator.

Accuracy and Usage :

The Gauss-Krüger system is beneficial for large-scale (detailed) maps due to its high accuracy over short distances. It is commonly used in civil engineering, cadastral mapping, and various geospatial applications within Germany and neighbouring countries.

Conversion to WGS84 :

Converting Gauss-Krüger coordinates to the more globally used WGS84 system (used in GPS) requires specific transformation parameters and sometimes complex algorithms due to the differences in ellipsoid and projection methods.

Software and Tools :

Various GIS software (like ArcGIS, QGIS) and online tools can perform these transformations. They typically require inputting the zone number, the easting, and the northing to transform the coordinates accurately.

History of the Gauss-Krüger Coordinate System

The Gauss-Krüger coordinate system has its roots in the early development of geodesy and map projection techniques in the 19th century. Here is a detailed history of the Gauss-Krüger coordinate system:

Carl Friedrich Gauss (1777-1855)

Contribution to Mathematics and Geodesy :

Carl Friedrich Gauss, a German mathematician and physicist, made significant contributions to many fields, including geodesy, the science of measuring and understanding the Earth’s geometric shape. Gauss developed the mathematical foundations for the projection that bears his name, the transverse Mercator projection, which is essential for creating accurate maps of regions with large north-south extents.

Transverse Mercator Projection :

Gauss’s work on the transverse Mercator projection provided a method to project the Earth’s surface onto a plane with minimal distortion over relatively small areas. This projection uses a cylinder rotated 90 degrees, touching the Earth along a chosen meridian.

Johann Heinrich Louis Krüger (1857-1923)

Refinement and Application :

Johann Heinrich Louis Krüger, a German geodesist, refined Gauss’s projection method and applied it to practical mapping needs. Krüger’s refinements improved the projection’s mathematical accuracy, making it more suitable for detailed surveying and mapping work.

Development of the Gauss-Krüger Coordinate System

Germany and Central Europe :

The Gauss-Krüger coordinate system was adopted primarily in Germany and other Central European countries for detailed topographic and cadastral mapping. The system divides the region into longitudinal zones, each 3° wide, with a central meridian. This minimises distortion and ensures high accuracy over small areas.

Ellipsoid Models :

The system uses specific ellipsoid models, such as the Bessel 1841 ellipsoid, which closely approximates the shape of the Earth in these regions.

Technical Features

False Easting and Northing :

The system applies a false easting and northing to ensure that all coordinates within a zone are positive. Typically, the central meridian is assigned a false easting of 500,000 meters, and the equator is assigned a false northing.

Zone-Based System :

Each zone has its coordinate system, reducing the complexity of calculations and distortions. The zones are numbered, usually starting from a prime meridian (e.g., 9°E) and increasing by 3° for each subsequent zone.

Modern Developments and Usage

Integration with Global Systems :

With the advent of global positioning systems (GPS) and the universal adoption of the WGS84 ellipsoid, many regions have transitioned to the UTM (Universal Transverse Mercator) system for broader compatibility. However, the Gauss-Krüger system is still used for specific applications requiring high precision and historical continuity in countries like Germany.

Software and Digital Mapping :

Modern GIS (Geographic Information Systems) software supports the Gauss-Krüger projection, allowing for easy conversion between coordinate systems and integration with global datasets.

The Gauss-Krüger coordinate system is a significant development in the history of cartography and geodesy. It combines the foundational work of Carl Friedrich Gauss and the practical refinements of Johann Heinrich Louis Krüger. Its precise and detailed approach to mapping has made it a valuable tool in Europe, particularly for topographic and cadastral mapping. While global systems like UTM are now more widely used, the Gauss-Krüger system remains integral to geospatial history and practice.

How do you convert Coordinates into UTM Coordinates?

Converting Gauss-Krüger coordinates to UTM (Universal Transverse Mercator) coordinates involves a few steps. Both systems are based on the transverse Mercator projection but use different parameters and zone definitions. Here’s a step-by-step guide on how to perform this conversion:

Step-by-Step Conversion Process

Identify the Gauss-Krüger Zone :

Determine the Gauss-Krüger zone of your coordinates. Gauss-Krüger zones are typically 3° wide.

Central Meridian of Gauss-Krüger Zone :

Each Gauss-Krüger zone has a central meridian, which is usually a multiple of 3° (e.g., 9°E, 12°E, 15°E, etc.).

Translate to Geodetic Coordinates (Latitude and Longitude) :

Convert the Gauss-Krüger coordinates (easting and northing) to geodetic coordinates (latitude and longitude). This requires:

• The ellipsoid parameters (e.g., Bessel 1841 for Germany).
• The false easting (usually 500,000 meters).
• Applying the inverse transverse Mercator projection.

Determine the UTM Zone :

Determine the appropriate UTM zone for the longitude obtained from the geodetic coordinates. UTM zones are 6° wide.

Convert to UTM Coordinates :

Convert the geodetic coordinates to UTM coordinates using:

• The WGS84 ellipsoid parameters (commonly used for UTM).
• The UTM zone central meridian.
• Applying the transverse Mercator projection to obtain the UTM easting and northing.

Example Conversion

Let’s walk through an example to make it more transparent.

Given :

Gauss-Krüger coordinates: Easting = 3550000 meters, Northing = 5800000 meters. Gauss-Krüger zone central meridian: 12°E (assuming it’s in zone 4)

Translate Gauss-Krüger to Latitude and Longitude :

Use the inverse Gauss-Krüger projection to convert (3550000, 5800000) to latitude and longitude. Due to the complexity of the inverse projection, this step typically requires software or detailed formulas.

Example Result :

Let’s assume the resulting geodetic coordinates are:

• Latitude: 52.0°N
• Longitude: 13.0°E

Determine UTM Zone :

Longitude 13.0°E falls in UTM zone 33U (UTM zones range from 1 to 60, each 6° wide).

Convert to UTM Coordinates :**

Use the transverse Mercator projection with WGS84 parameters and the central meridian of zone 33 (15°E) to convert latitude 52.0°N and longitude 13.0°E to UTM coordinates.

Online Tools:

Websites likehttps://epsg.io/ or other geospatial transformation tools can also perform these conversions.

Using Java:

To convert Gauss-Krüger coordinates to UTM coordinates in Java, you can use libraries such as Proj4j, which is a Java port of the “PROJ.4” library (https://de.wikipedia.org/wiki/PROJ.4) used for performing cartographic transformations. Here’s how you can perform the conversion step-by-step:

Add Proj4j Library to Your Project :

If you are using Maven, add the following dependency to yourpom.xml:

xmlCopy

<dependency><groupId>org.locationtech.proj4j</groupId><artifactId>proj4j</artifactId><version>1.1.0</version></dependency>

Define the Coordinate Reference Systems :

You will need to define the Gauss-Krüger and UTM coordinate reference systems.

Perform the Conversion :

Convert Gauss-Krüger coordinates to geodetic coordinates (latitude and longitude). Convert the geodetic coordinates to UTM coordinates.

Here is an example Java program that demonstrates this process:

javaCopy

importorg.locationtech.proj4j.CRSFactory;importorg.locationtech.proj4j.CoordinateReferenceSystem;importorg.locationtech.proj4j.ProjCoordinate;importorg.locationtech.proj4j.Projection;importorg.locationtech.proj4j.ProjectionFactory;importorg.locationtech.proj4j.ProjectionTransform;publicclassGaussKrugerToUTMConverter{publicstaticvoidmain(String[]args){// Create a CRSFactory instanceCRSFactoryfactory=newCRSFactory();// Define the Gauss-Krüger CRS (EPSG:31468 for zone 4)CoordinateReferenceSystemgaussKrugerCRS=factory.createFromName("EPSG:31468");// Define the UTM CRS (EPSG:32633 for UTM zone 33N)CoordinateReferenceSystemutmCRS=factory.createFromName("EPSG:32633");// Define the source coordinates in Gauss-Krüger (example values)doublegkEasting=3550000;doublegkNorthing=5800000;// Create a ProjCoordinate object for the input coordinatesProjCoordinategkCoord=newProjCoordinate(gkEasting,gkNorthing);// Create a ProjCoordinate object for the intermediate geodetic coordinatesProjCoordinategeoCoord=newProjCoordinate();// Create a ProjectionTransform object to convert from Gauss-Krüger to geodeticProjectionTransformgkToGeoTransform=newProjectionTransform(gaussKrugerCRS.getProjection(),ProjectionFactory.getGeographic());// Transform Gauss-Krüger to geodetic coordinates (longitude, latitude)gkToGeoTransform.transform(gkCoord,geoCoord);// Print the geodetic coordinates (longitude, latitude)System.out.println("Geodetic coordinates: Longitude = "+geoCoord.x+", Latitude = "+geoCoord.y);// Create a ProjCoordinate object for the final UTM coordinatesProjCoordinateutmCoord=newProjCoordinate();// Create a ProjectionTransform object to convert from geodetic to UTMProjectionTransformgeoToUtmTransform=newProjectionTransform(ProjectionFactory.getGeographic(),utmCRS.getProjection());// Transform geodetic coordinates to UTM coordinatesgeoToUtmTransform.transform(geoCoord,utmCoord);// Print the UTM coordinates (easting, northing)System.out.println("UTM coordinates: Easting = "+utmCoord.x+", Northing = "+utmCoord.y);}}

Dependencies :

Theproj4j library is added as a dependency to handle coordinate transformations.

Coordinate Reference Systems (CRS) :

The Gauss-Krüger CRS is defined usingEPSG:31468 for zone 4. The UTM CRS is defined usingEPSG:32633 for UTM zone 33N.

Transformation Process :

AProjCoordinate object is created for the input Gauss-Krüger coordinates. The Gauss-Krüger coordinates are transformed into geodetic coordinates (longitude and latitude). The geodetic coordinates are then transformed to UTM coordinates.

Notes :

Ensure theepsg codes are correct for your specific regions and projections. The actual geodetic transformation can be complex due to ellipsoid differences, so precision might require specific parameters and fine-tuning.

]]>Navigationhttps://svenruppert.com/posts/cwe-787-the-bird-eye-view-for-java-developers/Wed, 15 May 2024 12:19:10 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cwe-787-the-bird-eye-view-for-java-developers/The term “CWE-787: Out-of-bounds Write " likely refers to a specific security vulnerability or error in software systems. Let’s break down what it means: Out-of-bounds Write : This is a type of vulnerability where a program writes data outside the boundaries of pre-allocated fixed-length buffers. This can corrupt data, crash the program, or lead to the execution of malicious code.<![CDATA[

The term “CWE-787: Out-of-bounds Write " likely refers to a specific security vulnerability or error in software systems. Let’s break down what it means:

Out-of-bounds Write : This is a type of vulnerability where a program writes data outside the boundaries of pre-allocated fixed-length buffers. This can corrupt data, crash the program, or lead to the execution of malicious code.

CWE-787 : This is a specific identifier for the vulnerability. Identifiers like these are often used in vulnerability databases or bug-tracking systems. They help uniquely identify and reference a particular issue.

1. Explanation of Out-of-bounds Write
2. Impact of Out-of-bounds Write
3. Examples and Mitigation
   1. Example in C:
   2. Mitigation:
   3. Example of Safe Code:
4. And how is it done in Java?
   1. Understanding Out-of-bounds Write in Java
   2. Example of Out-of-bounds Write
   3. Mitigation Strategies
   4. Corrected Example
5. Common Causes of Out-of-bounds Write in Java
6. Advanced Example
   1. Mitigated Version
7. CWE 787 - Based on OffHeap techniques in Java 8
   1. Off-heap Memory Management in Java
   2. Example of Out-of-bounds Write with Off-heap Memory
      1. Usingsun.misc.Unsafe
      2. Mitigating Out-of-bounds Write with Off-heap Memory
      3. Corrected Example withsun.misc.Unsafe
      4. Usingjava.nio.ByteBuffer
   3. Handling Larger Data Structures
      1. Example withIntBuffer
      2. Explanation
8. OffHeap usage in Java 21
   1. Critical Features for Off-Heap Usage in Java 21
   2. Foreign Function & Memory API
   3. Example Using Foreign Function & Memory API
   4. EnhancedByteBuffer Operations
   5. Memory Segments and Access VarHandles
      1. Example Using Memory Segments
9. Differences in OffHeap usage between Java 17 and Java 21
   1. Foreign Function & Memory API
   2. Enhanced ByteBuffer Operations
   3. Memory Segments and VarHandles
   4. Summary of Differences

Explanation of Out-of-bounds Write

An out-of-bounds write occurs when a program writes data past the end, or before the beginning, of the buffer it was allocated. This usually happens due to programming errors such as:

• Incorrectly calculating buffer sizes.
• Failing to check the bounds before writing data.
• Using unsafe functions that do not perform bounds checking.

Impact of Out-of-bounds Write

The consequences of an out-of-bounds write can be severe, including:

Data Corruption : Overwriting adjacent memory locations can corrupt other data.

Program Crashes : Writing to illegal memory addresses can cause the program to crash.

Security Vulnerabilities : Attackers can exploit these vulnerabilities to execute arbitrary code, leading to potential breaches, data leaks, or system compromise.

Examples and Mitigation

Example in C:

xmlCopy

#include<string.h>#include<stdio.h>int main() {    char buffer[10];    strcpy(buffer, "This string is too long for the buffer");    printf("%s\n", buffer);    return 0;}

In this example, the string copied intobuffer exceeds its allocated size, causing an out-of-bounds write.

Mitigation:

Bounds Checking : Ensure that any write operations do not exceed the allocated buffer size.

Use Safe Functions : Utilise safer library functions likestrncpy instead ofstrcpy.

Static Analysis : Use static analysis tools to detect out-of-bounds writes during development.

Runtime Protection : Employ runtime protections such as address space layout randomisation (ASLR) and stack canaries.

Example of Safe Code:

xmlCopy

#include<string.h>#include<stdio.h>int main() {    char buffer[10];    strncpy(buffer, "This string is too long for the buffer", sizeof(buffer) - 1);    buffer[sizeof(buffer) - 1] = '\0';  // Ensure null termination    printf("%s\n", buffer);    return 0;}

In this corrected version,strncpy ensures that no more than the allocated size ofbuffer is written, and null-terminating the string prevents buffer overflow.

Understanding and preventing out-of-bounds writes is crucial in developing secure and stable software. Proper coding practices, safe functions, and security measures can mitigate these vulnerabilities effectively.

And how is it done in Java?

Out-of-bounds write vulnerabilities are less common in Java than in languages like C or C++ due to built-in safety features, such as automatic bounds checking for arrays. However, certain situations can still lead to such vulnerabilities, often related to misusing arrays or improper handling of indices.

Understanding Out-of-bounds Write in Java

In Java, out-of-bounds write errors typically occur when you try to access an array or a list with an index that is either negative or greater than or equal to the size of the array or list. This usually results in anArrayIndexOutOfBoundsException.

Example of Out-of-bounds Write

javaCopy

publicclassOutOfBoundsWriteExample{    publicstaticvoidmain(String[]args){        int[]numbers=newint[5];        for(inti=0;i<=numbers.length;i++){// Off-by-one error            numbers[i]=i*2;// This will throw an ArrayIndexOutOfBoundsException        }    }}

In this example, the loop runs from0 tonumbers.length, one past the last valid index. This causes anArrayIndexOutOfBoundsException.

Mitigation Strategies

Proper Bounds Checking : Always ensure that any access to arrays or lists is within valid bounds.

Enhanced for Loop : Use enhanced for loops when possible to avoid index errors.

Libraries and Functions : Use Java’s standard libraries and methods that handle bounds checking automatically.

Corrected Example

javaCopy

publicclassOutOfBoundsWriteExample{    publicstaticvoidmain(String[]args){        int[]numbers=newint[5];        for(inti=0;i<numbers.length;i++){// Correct bounds            numbers[i]=i*2;        }        for(intnumber:numbers){            System.out.println(number);        }    }}

In this corrected version, the loop runs from0 tonumbers.length—1, which prevents theArrayIndexOutOfBoundsException.

Common Causes of Out-of-bounds Write in Java

Off-by-one Errors : These occur when loops iterate one time too many or too few.

Incorrect Index Calculation : When the calculation of an index is erroneous due to logic errors.

Manual Array Management : Errors often happen when manipulating arrays directly rather than using collections likeArrayList.

Advanced Example

Consider a scenario where an attempt to manage a buffer directly leads to an out-of-bounds write:

javaCopy

publicclassBufferOverflowExample{    publicstaticvoidmain(String[]args){        int[]buffer=newint[10];        try{            writeToBuffer(buffer,10,42);// Trying to write outside the buffer        }catch(ArrayIndexOutOfBoundsExceptione){            System.out.println("Caught exception: "+e);        }    }    publicstaticvoidwriteToBuffer(int[]buffer,intindex,intvalue){        buffer[index]=value;// Potential out-of-bounds write    }}

In this example,writeToBuffer is called with an index that is out of bounds, leading to anArrayIndexOutOfBoundsException.

Mitigated Version

javaCopy

publicclassBufferOverflowExample{    publicstaticvoidmain(String[]args){        int[]buffer=newint[10];        try{            writeToBuffer(buffer,10,42);// Trying to write outside the buffer        }catch(IllegalArgumentExceptione){            System.out.println("Caught exception: "+e);        }    }    publicstaticvoidwriteToBuffer(int[]buffer,intindex,intvalue){        if(index<0||index>=buffer.length){            thrownewIllegalArgumentException("Index out of bounds: "+index);        }        buffer[index]=value;    }}

Here,writeToBuffer checks the index before writing to the buffer, preventing an out-of-bounds write.

While Java provides many safeguards against out-of-bounds writes, developers must still practice diligent bounds checking and utilise safe coding practices to prevent these errors. Handling array indices and leveraging Java’s robust standard libraries can help maintain secure and stable code.

CWE 787 - Based on OffHeap techniques in Java 8

“off-heap” in Java refers to memory allocated outside the Java heap, typically managed by native code. Working with off-heap memory can provide performance benefits, especially for large data sets or when interacting with native libraries. Still, it also introduces risks such as out-of-bounds writes.

Off-heap Memory Management in Java

Java provides several ways to work with off-heap memory, including thesun.misc.Unsafe class andjava.nio.ByteBuffer. Thesun.misc.Unsafe class allows low-level, unsafe operations, and should be used with caution. Thejava.nio.ByteBuffer class is safer but requires careful bounds checking.

Example of Out-of-bounds Write with Off-heap Memory

Usingsun.misc.Unsafe

javaCopy

importsun.misc.Unsafe;importjava.lang.reflect.Field;publicclassOffHeapExample{    privatestaticfinalUnsafeunsafe;    privatestaticfinallongBYTE_ARRAY_BASE_OFFSET;    static{        try{            Fieldfield=Unsafe.class.getDeclaredField("theUnsafe");            field.setAccessible(true);            unsafe=(Unsafe)field.get(null);            BYTE_ARRAY_BASE_OFFSET=unsafe.arrayBaseOffset(byte[].class);        }catch(Exceptione){            thrownewRuntimeException(e);        }    }    publicstaticvoidmain(String[]args){        longsize=10;        longmemoryAddress=unsafe.allocateMemory(size);        try{            for(inti=0;i<=size;i++){// Intentional off-by-one error                unsafe.putByte(memoryAddress+i,(byte)i);            }        }finally{            unsafe.freeMemory(memoryAddress);        }    }}

In this example, the loop iterates one time too many, causing an out-of-bounds write wheni equalssize.

Mitigating Out-of-bounds Write with Off-heap Memory

Bounds Checking : Always ensure that memory accesses are within the allocated bounds.

Abstraction : Use higher-level abstractions that handle bounds checking automatically, such asjava.nio.ByteBuffer.

Corrected Example withsun.misc.Unsafe

javaCopy

importsun.misc.Unsafe;importjava.lang.reflect.Field;publicclassOffHeapExample{    privatestaticfinalUnsafeunsafe;    privatestaticfinallongBYTE_ARRAY_BASE_OFFSET;    static{        try{            Fieldfield=Unsafe.class.getDeclaredField("theUnsafe");            field.setAccessible(true);            unsafe=(Unsafe)field.get(null);            BYTE_ARRAY_BASE_OFFSET=unsafe.arrayBaseOffset(byte[].class);        }catch(Exceptione){            thrownewRuntimeException(e);        }    }    publicstaticvoidmain(String[]args){        longsize=10;        longmemoryAddress=unsafe.allocateMemory(size);        try{            for(inti=0;i<size;i++){// Correct bounds                unsafe.putByte(memoryAddress+i,(byte)i);            }        }finally{            unsafe.freeMemory(memoryAddress);        }    }}

Here, the loop correctly iterates from0 tosize - 1, preventing the out-of-bounds write.

Usingjava.nio.ByteBuffer

A safer alternative involves usingjava.nio.ByteBuffer for off-heap memory management. ByteBuffer provides bounds checking, reducing the risk of out-of-bounds writes.

javaCopy

importjava.nio.ByteBuffer;publicclassOffHeapByteBufferExample{    publicstaticvoidmain(String[]args){        intsize=10;        ByteBufferbuffer=ByteBuffer.allocateDirect(size);        for(inti=0;i<size;i++){// Correct bounds            buffer.put((byte)i);        }        buffer.flip();// Prepare the buffer for reading        while(buffer.hasRemaining()){            System.out.println(buffer.get());        }    }}

In this example,ByteBuffer manages off-heap memory safely with built-in bounds checking.

Managing off-heap memory in Java can offer performance benefits but requires careful handling to avoid out-of-bounds writes. Usingsun.misc.Unsafe provides powerful capabilities but comes with significant risks. For safer memory management,java.nio.ByteBuffer is recommended due to its automatic bounds checking. Always perform thorough bounds checking and prefer higher-level abstractions to ensure memory safety. But there are changes in newer Java releases.

Handling Larger Data Structures

For more complex data structures, you can use methods likeasIntBuffer,asLongBuffer, etc., which provide views of the buffer with different data types.

Example withIntBuffer

javaCopy

importjava.nio.ByteBuffer;importjava.nio.IntBuffer;publicclassOffHeapIntBufferExample{    publicstaticvoidmain(String[]args){        intsize=10;        // Allocate off-heap memory        ByteBufferbyteBuffer=ByteBuffer.allocateDirect(size*Integer.BYTES);        IntBufferintBuffer=byteBuffer.asIntBuffer();        // Writing data to the buffer        for(inti=0;i<size;i++){            intBuffer.put(i);// Proper bounds checking is inherent        }        // Preparing the buffer to read the data        intBuffer.flip();        // Reading data from the buffer        while(intBuffer.hasRemaining()){            System.out.println(intBuffer.get());        }    }}

Explanation

Allocation :ByteBuffer.allocateDirect(size * Integer.BYTES) allocates a direct buffer with enough space forsize integers.

IntBuffer View :byteBuffer.asIntBuffer() creates anIntBuffer view of theByteBuffer, directly allowing you to work with integers.

Writing and Reading : Theput andget methods ofIntBuffer are used for writing and reading integers, with bounds checking.

Usingjava.nio.ByteBuffer for off-heap memory management in Java provides a safer alternative tosun.misc.Unsafe, ensuring automatic bounds checking and reducing the risk of out-of-bounds writes. Following the outlined examples and strategies, you can effectively manage off-heap memory while maintaining code safety and integrity.

OffHeap usage in Java 21

Java 21 introduces several enhancements and features for off-heap memory management, focusing on improved performance, safety, and ease of use. While the core approach usingjava.nio.ByteBuffer remains prevalent, new features and improvements in the Java ecosystem help with off-heap usage.

Critical Features for Off-Heap Usage in Java 21

Foreign Function & Memory API (Preview): Java 21 includes a preview of the Foreign Function & Memory API, designed to facilitate interactions with native code and memory. This API allows for safe and efficient off-heap memory access and manipulation.

EnhancedByteBuffer Operations : TheByteBuffer class has been optimised and extended with more operations, making it more efficient for handling off-heap memory.

Memory Segments and Access VarHandles : Java 21 introduces memory segments and access varhandles for structured memory access. This provides a safer and more structured way to handle off-heap memory.

Foreign Function & Memory API

The Foreign Function & Memory API allows Java programs to interoperate with native libraries and manage off-heap memory more safely and efficiently. This API includes classes such asMemorySegment,MemoryAddress, andMemoryLayout to represent and manipulate memory.

Example Using Foreign Function & Memory API

Here is an example of how you might use the Foreign Function & Memory API to allocate and manipulate off-heap memory:

javaCopy

importjdk.incubator.foreign.*;publicclassOffHeapExample{    publicstaticvoidmain(String[]args){// Allocate 100 bytes of off-heap memory        try(MemorySegmentsegment=MemorySegment.allocateNative(100)){            MemoryAddressbaseAddress=segment.baseAddress();            // Write data to off-heap memory            for(inti=0;i<100;i++){                segment.set(ValueLayout.JAVA_BYTE,i,(byte)i);            }            // Read data from off-heap memory            for(inti=0;i<100;i++){                bytevalue=segment.get(ValueLayout.JAVA_BYTE,i);                System.out.println("Value at index "+i+": "+value);            }        }    }}

EnhancedByteBuffer Operations

WhileByteBuffer remains a crucial class for off-heap memory management, Java 21’s performance improvements make it even more suitable for high-performance applications.

Memory Segments and Access VarHandles

The new memory segment API provides a safer and more structured way to handle off-heap memory, replacing the need forsun.misc.Unsafe.

Example Using Memory Segments

javaCopy

importjdk.incubator.foreign.*;publicclassMemorySegmentExample{    publicstaticvoidmain(String[]args){// Allocate 10 integers worth of off-heap memory        try(MemorySegmentsegment=MemorySegment.allocateNative(10*4)){            VarHandleintHandle=MemoryHandles.varHandle(int.class,ByteOrder.nativeOrder());            // Writing data to the memory segment            for(inti=0;i<10;i++){                intHandle.set(segment.baseAddress().addOffset(i*4),i);            }            // Reading data from the memory segment            for(inti=0;i<10;i++){                intvalue=(int)intHandle.get(segment.baseAddress().addOffset(i*4));                System.out.println("Value at index "+i+": "+value);            }        }    }}

Java 21 enhances off-heap memory management with new and improved features such as the Foreign Function & Memory API and enhancedByteBuffer operations. These features provide safer, more efficient, and more flexible ways to interact with off-heap memory. By leveraging these new capabilities, developers can write high-performance, memory-safe applications that interact seamlessly with native libraries and off-heap memory.

Differences in OffHeap usage between Java 17 and Java 21

Java 21 introduces several enhancements and new features for off-heap memory management compared to Java 17. Here’s a detailed comparison highlighting the key differences:

Foreign Function & Memory API

Java 17 :

Incubator Stage : The Foreign Function & Memory API was in an incubator stage, which was experimental and subject to change. The API provided the foundations for off-heap memory access and foreign function calls but lacked maturity and stability.

Java 21 :

Preview Stage : The Foreign Function & Memory API is now in a preview stage, indicating it has become more stable and mature. This API allows for safer, more efficient off-heap memory management and interoperability with native libraries.

MemorySegment and MemoryAddress : These abstractions provide structured and safe access to off-heap memory, reducing the risks associated with manual memory management.

MemoryLayout and VarHandles : These additions enable developers to describe the layout of memory segments and access them in a type-safe manner using varhandles.

Enhanced ByteBuffer Operations

Java 17 :

Basic Operations : TheByteBuffer class in Java 17 supports basic operations for off-heap memory allocation and access throughallocateDirect.

Performance : While effective, the performance optimisations were less advanced than in Java 21.

Java 21 :

Performance Improvements : Java 21 includes performance optimisations forByteBuffer, making it more efficient for high-performance applications.

Extended Operations : Additional operations and methods may have been introduced or optimised to enhance functionality and ease of use.

Memory Segments and VarHandles

Java 17 :

Limited Usage : Memory segments and varhandles were part of the incubator Foreign Memory Access API, not widely adopted or stable.

Java 21 :

Stabilised and Enhanced : These concepts have been further developed and stabilised, providing a more robust way to handle off-heap memory.

Structured Access : Memory segments offer a structured way to allocate, access, and manage off-heap memory, reducing risks and improving safety.

VarHandles : Provide a type-safe mechanism to manipulate memory, akin to low-level pointers but with safety checks.

Summary of Differences

API Maturity : The Foreign Function & Memory API has evolved from an incubator stage in Java 17 to a preview stage in Java 21, providing a more stable and feature-rich environment.

Performance : Java 21 offers performance improvements and more optimised operations forByteBuffer and other memory-related classes.

Memory Management : Java 21 introduces more structured and safer ways to handle off-heap memory with memory segments and varhandles.

Safety and Efficiency : Java 21’s improvements emphasise safety and efficiency, reducing the risks associated with manual off-heap memory management.

These advancements in Java 21 enhance the capabilities and safety of off-heap memory management, making it a more robust choice for developers needing high-performance memory operations.

]]>JavaSecure Coding PracticesSecurityhttps://svenruppert.com/posts/mastering-secure-error-handling-in-java-best-practices-and-strategies/Tue, 07 May 2024 12:43:21 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/mastering-secure-error-handling-in-java-best-practices-and-strategies/What is ErrorHandling? Error handling refers to the programming practice of anticipating, detecting, and responding to exceptions or errors in software during its execution. Errors may occur for various reasons, such as invalid user inputs, hardware failures, or bugs in the code. Proper error handling helps ensure that the program can handle such situations gracefully by resolving the Error, compensating for it, or failing safely.<![CDATA[

What is ErrorHandling?

Error handling refers to the programming practice of anticipating, detecting, and responding to exceptions or errors in software during its execution. Errors may occur for various reasons, such as invalid user inputs, hardware failures, or bugs in the code. Proper error handling helps ensure that the program can handle such situations gracefully by resolving the Error, compensating for it, or failing safely.

1. What is ErrorHandling?
2. How is error handling done in Java?
   1. Types of Exceptions
   2. Exception Handling Mechanisms
      1. Try-Catch Block
      2. Throws Keyword
      3. Throw Keyword
   3. Custom Exceptions
   4. Best Practices
3. Why is Error handling necessary for security?
4. How to avoid lousy Error Handling?
   1. Information Leakage through Error Messages
   2. Denial of Service (DoS)
   3. Exception Handling Overhead
   4. Uncaught Exceptions
   5. Improper Use of Checked and Unchecked Exceptions
   6. Security Decisions Based on Exception Handling
   7. Error Handling and Code Injection Risks

Here are some critical aspects of error handling:

Detection : Identifying situations that may lead to errors.

Intervention : Implement strategies to handle the Error once it is detected. This may include logging the Error for further analysis, notifying users or administrators, and attempting recovery.

Recovery : Trying to continue the program’s execution by bypassing the Error or correcting the error condition.

Graceful Degradation : If recovery is not possible, the system may continue operating with reduced functionality.

Fail-Safe Operations : Ensuring that the system fails in a way that does not cause harm or excessive inconvenience to the user.

Programming languages and environments typically provide mechanisms to handle errors, such as Java, Python, or C++ exceptions. Handling these exceptions allows developers to write more robust and reliable software.

How is error handling done in Java?

Error handling in Java is primarily accomplished through the use of exceptions. Exceptions are events that can alter the normal flow of program execution when an error or other unusual condition occurs. Java provides a robust and structured way to catch and handle these exceptions, making it possible to manage errors effectively. Here’s how error handling is typically done in Java:

Types of Exceptions

Java categorises exceptions into two main types:

Checked Exceptions : These exceptions must be explicitly caught or declared in the method where they might be thrown. They are checked at compile time, meaning the compiler requires the programmer to handle them, typically throughtry-catch blocks or by declaring the Exception in the method signature withthrows.

Unchecked Exceptions : These include runtime exceptions and errors and do not need to be explicitly handled. Runtime exceptions typically result from programming errors such as logic mistakes, improper use of an API, etc. Errors are used by the Java runtime to indicate serious problems that a reasonable application should not try to catch, likeOutOfMemoryError.

Exception Handling Mechanisms

Try-Catch Block

The primary exception mechanism is thetry-catch block. Here’s how it works:

javaCopy

try{    //code that might throw an exception}catch(ExceptionTypename){    //code that handles the Exception}finally{    //code that executes after try-catch, regardless of whether an exception was thrown}

**try**: The block that contains the code which might throw an exception.

**catch**: This block catches the Exception thrown by thetry block. Multiple catch blocks can handle different exceptions.

**finally**: This block is optional and executes after the try-and-catch blocks. It executes regardless of whether an exception was thrown or caught. It’s typically used for cleanup activities (like closing file streams).

Throws Keyword

If a method is capable of causing an exception that it does not handle, it must declare this Exception with thethrows keyword in its method signature:

javaCopy

publicvoidmyMethod()throwsIOException{    //code that might throw an IOException}

Throw Keyword

Java also allows you to throw an exception explicitly using thethrow keyword. This is often used to throw a custom exception or re-throw a caught exception after some specific handling:

javaCopy

**thrownewException("This is an error message");**

Custom Exceptions

You can create your exception types by extending theException class (for checked exceptions) or theRuntimeException class (for unchecked exceptions). This is useful for creating application-specific exceptions that can carry more contextual information about the Error:

javaCopy

publicclassMyCustomExceptionextendsException{    publicMyCustomException(Stringmessage){        super(message);    }}

Best Practices

- Catch the most specific Exception first before the more general ones.

- Avoid catchingThrowable,Exception, orRuntimeException unless necessary, as it can hide bugs.

- Always clean up after handling an exception, either in afinally block or using try-with-resources for AutoCloseable resources.

By following these structured approaches, Java programs can handle errors gracefully, making applications more robust, secure, and user-friendly.

Why is Error handling necessary for security?

Error handling is critically important for security for several reasons, as it directly impacts the robustness and resilience of a system. Here are some key ways in which effective error handling enhances security:

Preventing Information Disclosure : Poor error handling can unintentionally leak sensitive information. For example, a detailed error message could reveal system details, such as file paths, database schema, or software versions, which attackers could exploit. Proper error handling should mask or generalise error messages shown to users while logging detailed information securely for debugging purposes by administrators.

Avoiding Denial of Service (DoS) : Software that does not gracefully handle errors can crash or become unresponsive when faced with unexpected conditions, making it vulnerable to DoS attacks. Proper error handling ensures that the application can continue to operate or fail safely without affecting overall availability.

Mitigating Injection Flaws : Applications that fail to handle errors properly are often vulnerable to injection attacks, such as SQL injection, where attackers exploit error messages to glean insights about the database structure. By securely handling errors, applications can prevent such attacks from being successful.

Ensuring Data Integrity : Effective error handling can help maintain data integrity by checking for errors during data processing and preventing corrupt or partial data from being saved to the database. This is crucial in avoiding scenarios where erroneous data could lead to security vulnerabilities or functional errors.

Managing Untrusted Data and Inputs : By handling errors properly, applications can safely deal with untrusted inputs that might otherwise cause failures or security breaches. For instance, if an application tries to process a considerable file and fails due to resource limitations, properly handling this condition prevents it from being used as a vector for attacks.

Upholding Compliance and Trust : Many regulatory standards require robust error handling as part of compliance mandates. Failing to implement proper error handling can lead to compliance failures and legal liabilities. Moreover, a well-handled error scenario can maintain user trust, whereas repeated failures or uninformative error messages can erode it.

Overall, robust error handling is essential in designing secure systems. It prevents specific security vulnerabilities and contributes to the software’s overall resilience and reliability.

How to avoid lousy Error Handling?

When not appropriately implemented, error handling in Java can have several security implications. Here’s a detailed look at potential issues and how to mitigate them:

Information Leakage through Error Messages

Problem : Detailed error messages can inadvertently reveal system valuable information to attackers. For example, stack traces exposed to users might include information about the software architecture, third-party libraries being used, database information, or specific methods in the code.

Mitigation : Avoid sending detailed error messages to the client or end-user. Instead, log these details internally, where developers or system administrators can use them for debugging. Show users generic, user-friendly error messages that do not disclose sensitive information.

Denial of Service (DoS)

Problem : If exceptions are not handled properly, an attacker might be able to exploit this by sending requests that they know will cause exceptions, potentially leading the application to crash or become unresponsive.

Mitigation : Implement robust Exception handling that isolates error conditions and ensures the application can recover gracefully. Use techniques like input validation to prevent problematic data from causing unexpected behaviour.

Exception Handling Overhead

Problem : Creating, throwing, and catching exceptions can be resource-intensive, especially if the exceptions involve stack trace generation. If an application generates many exceptions as part of normal operations, it can lead to performance issues and resource exhaustion.

Mitigation : Optimise exceptions by avoiding them in standard flow control and using them only for exceptional conditions. Consider using return codes or validation methods to handle common, expected conditions.

Uncaught Exceptions

Problem : The application can miss exceptions, causing the program to terminate unexpectedly, leading to potential denial of service and other stability issues.

Mitigation : Ensure that all parts of the application are covered by adequate exception handling, and use global exception handlers as a last resort to catch and log any unhandled exceptions.

Improper Use of Checked and Unchecked Exceptions

Problem : Misusing checked and unchecked exceptions can lead to poorly designed code that hides errors (unchecked) or burdens the developer with too many error-handling responsibilities (checked), potentially leading to ignored exceptions.

Mitigation : Use checked exceptions for recoverable conditions and where you want to enforce the caller’s handling. Use unchecked exceptions for programming errors that should not be handled programmatically.

Security Decisions Based on Exception Handling

Problem : Relying on exception handling to make security decisions, such as authentication or validation, can lead to logic errors and potential security vulnerabilities.

Mitigation : Design security controls independently of the exception-handling mechanism. Ensure that security decisions are made based on secure, well-established practices rather than the presence or absence of exceptions.

Error Handling and Code Injection Risks

Problem : Poorly handled errors can lead to injection vulnerabilities if the output from exceptions is not properly sanitised before being displayed to the user.

Mitigation : Always sanitise any dynamic content displayed to users, including error messages, to prevent cross-site scripting (XSS) and other injection attacks.

Developers can significantly enhance Java applications’ security posture by understanding and addressing these security implications. Effective error handling prevents crashes, improves user experience, and strengthens the application’s resistance to malicious attacks.

]]>JavaSecure Coding Practiceshttps://svenruppert.com/posts/decoding-the-logs-essential-insights-for-effective-software-debugging/Mon, 06 May 2024 21:12:46 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/decoding-the-logs-essential-insights-for-effective-software-debugging/Logging is essential to software development, recording information about the software’s operation. This can help developers understand the system’s behaviour, troubleshoot issues, and monitor the system in production. Here’s a basic overview of logging in software development:<![CDATA[

Logging is essential to software development, recording information about the software’s operation. This can help developers understand the system’s behaviour, troubleshoot issues, and monitor the system in production. Here’s a basic overview of logging in software development:

1. Purpose of Logging
   1. Debugging and Troubleshooting:
   2. Monitoring System Health:
   3. Audit Trails:
   4. Security Analysis:
   5. Operational Intelligence:
   6. Regulatory Compliance:
   7. Notification of Important Events:
2. What to Log
   1. Errors and Exceptions
   2. System Events
   3. User Actions
   4. Performance Metrics
   5. Security Events
   6. Warnings
3. Logging Levels
   1. DEBUG
   2. INFO
   3. NOTICE
   4. WARNING
   5. ERROR
   6. CRITICAL
   7. ALERT
   8. EMERGENCY
   9. Choosing the Right Level
4. Logging Best Practices
   1. Use Established Logging Frameworks
   2. Implement Appropriate Log Levels
   3. Ensure Logs Are Contextual and Informative
   4. Centralise Log Management
   5. Secure and Protect Log Data
   6. Regularly Monitor and Analyse Logs
   7. Manage Log Volume
   8. Automate Log Analysis
   9. Implement Log Retention Policies
   10. Use Asynchronous and Non-blocking Logging
   11. Standardise Logs Across Services
5. Tools for Log Management
   1. ELK Stack
   2. Splunk
   3. Graylog
   4. Fluentd
   5. Datadog
   6. Loggly
   7. Papertrail
   8. Prometheus and Grafana
   9. Choosing the Right Tool
6. Common Pitfalls
   1. Over-Logging
   2. Under-Logging
   3. Ignoring Logs
   4. Not Protecting Log Information
   5. Inconsistent Logging
   6. Lack of Context in Logs
   7. Blocking or Synchronous Logging
   8. Poor Log Management and Retention Practices
   9. Failing to Plan for Log Scalability

Purpose of Logging

The purpose of logging in software development is multifaceted, encompassing several vital aspects that collectively improve software systems’ operability, security, and maintainability. Here are the primary purposes of logging:

Debugging and Troubleshooting:

Logging gives developers detailed insights into the application’s behaviour, allowing them to trace the steps leading up to errors, exceptions, and other anomalous behaviours. This is invaluable for debugging and can significantly reduce the time needed to diagnose and fix issues.

Monitoring System Health:

Logs can provide real-time information about the health of an application, including performance metrics such as response times, throughput, and resource utilisation. This helps proactively manage system performance and identify potential issues before they affect users.

Audit Trails:

Logging creates an audit trail in many applications, particularly those that handle financial transactions, sensitive data, or personal information. This helps track user actions and system changes, which is crucial for compliance with regulations and standards (such as GDPR, HIPAA, or SOX).

Security Analysis:

Logs can detect and alert potential security incidents. For instance, a high number of failed login attempts could indicate a brute-force attack. Logs also help in the forensic analysis after an incident, enabling the understanding of how the security breach occurred and the extent of the impact.

Operational Intelligence:

Analysing logs, businesses can gain insights into user behaviour, system usage patterns, and operational bottlenecks. This information can guide decisions on system improvements, user support, and new feature development.

Regulatory Compliance:

Logging is not just an operational tool but a legal requirement for many industries. Logs must be maintained to demonstrate compliance with various regulatory frameworks and show that the system performs correctly and securely.

Notification of Important Events:

Logging systems can be configured to send alerts when critical system events occur, such as system outages, significant performance degradation, or other critical issues that require immediate attention.

Logging is an essential discipline in software development that enhances visibility into applications, ensuring they run smoothly, securely, and in compliance with legal and operational requirements. It also helps developers and IT professionals effectively manage, diagnose, and optimise their software environments.

What to Log

Deciding what to log in a software system is crucial for ensuring you gather enough information to be helpful in debugging, monitoring, and compliance without overwhelming the system or the teams that need to parse through the logs. Here are key categories and specific elements you should consider logging:

Errors and Exceptions

Critical Failures : Any errors that cause a part of your system to fail or potentially disrupt service.

Exceptions : Catch and log exceptions with stack traces and context to understand why they occurred.

System Events

Startups and Shutdowns : Record when your system or service starts and stops.

Configuration Changes : Log any changes in a system configuration that might affect behaviour or performance.

Scheduled Tasks : Record when scheduled tasks begin and end, especially if they’re crucial to system functionality.

User Actions

Logins and Logouts : Tracking user sessions can help detect unauthorised access and understand user engagement.

Important Transactions : Logging these activities is vital, especially in systems handling payments, orders, or sensitive operations.

Access to Sensitive Data : Log in when users access sensitive information to comply with privacy laws and regulations.

Performance Metrics

Response Times : Log how long it takes to respond to user requests.

System Utilisation : Include metrics on CPU, memory, disk, and network usage.

Service Availability : Record any downtime or interruptions in service.

Security Events

Failed Logins : An excessive number of failed login attempts can indicate a brute-force attack.

Permission Changes : Track user permissions changes, especially for administrative access users.

Security Breaches : Log any detected breaches or potential security threats.

Warnings

Resource Limits : Warnings when resources (e.g., memory, disk space) run low.

Deprecations : Log usage of deprecated APIs or features that will be removed in future.

Logging Levels

Log levels are a way to categorise entries in your logs based on their importance and the detail of information they provide. These levels help filter logs, like debugging, monitoring, and alerting. Here’s a breakdown of expected log levels used in many logging systems, from the most verbose to the least:

DEBUG

Purpose : This is the most detailed level for total diagnostic output. It includes valuable information for developers during development and debugging to understand precisely what the system is doing.

Use Case : You would enable DEBUG logging when trying to solve a tricky bug or when you need a detailed trace of how data flows through your system.

INFO

Purpose : General information about the application’s operation. These entries should be informative and relevant to users or system administrators monitoring the healthy functioning of the application.

Use Case : Routine operations like user logins, SQL logs, and other operational milestones.

NOTICE

Purpose : Important runtime events that are not necessarily errors but may require attention or be significant in system auditing or analysis.

Use Case : Deprecation warnings, minor configuration issues, or other messages that don’t require immediate action but should be noted for potential future relevance.

WARNING

Purpose : Indicative of something unexpected or a potential problem in the future. However, the application can continue running.

Use Case : Recoverable malfunctions, such as retrying operations, missing secondary data, or running with default values due to missing configuration.

ERROR

Purpose : This indicator indicates issues that are of immediate concern, as they may hinder system operations or result in a partial application failure.

Use Case : Runtime errors, inability to access a necessary resource, exceptions that are handled but disrupt regular operation.

CRITICAL

Purpose : Very serious issues that might cause the application to terminate or affect essential functionality.

Use Case : Critical conditions include data loss scenarios, out-of-memory, or data corruption.

ALERT

Purpose : A step above CRITICAL, requiring immediate attention. This typically involves issues that need to be fixed immediately to prevent stoppage or significant damage to operations.

Use Case : Breaches in security, system components going down, loss of connectivity.

EMERGENCY

Purpose : This is the highest level, used when the system is unusable or major parts of the system are non-functional.

Use Case : Complete system outage, catastrophic failure, or anything that requires immediate and urgent attention to prevent harm or significant disruption.

Choosing the Right Level

When choosing which level to log in, consider the following:

• The audience for the logs : Developers, system administrators, end-users, or auditors.

• The environment : Development, testing, staging, or production.

• The severity of the event : The impact on the system’s functionality and the user’s experience.

Proper use of log levels allows you to control the verbosity of logging output, making the logs both manageable and meaningful. This control is significant in production environments where excessive logging can lead to performance degradation and increased log storage and management costs.

Logging Best Practices

Adopting best practices in logging is crucial for creating a robust and effective logging strategy. Here are several fundamental guidelines to help ensure your logging system is both efficient and valuable:

Use Established Logging Frameworks

Why : Leverage well-maintained and community-tested libraries to handle routine logging tasks. Examples include Log4j for Java, Serilog for .NET, Winston for Node.js, and Python’s built-inlogging module.

Benefit : These frameworks support features like log rotation, different logging levels, and integration with various logging backends.

Implement Appropriate Log Levels

Why : Differentiate log messages according to severity levels (DEBUG, INFO, WARNING, ERROR, etc.).

Benefit : Allows fine-grained control over which log entries are output based on the current runtime environment, helping to reduce noise in production logs and focus on relevant data.

Ensure Logs Are Contextual and Informative

Why : Log messages should include enough context to be understood independently. Context can consist of timestamps, user IDs, session IDs, and other relevant details.

Benefit : Makes troubleshooting easier by clarifying when and where events occur, especially in distributed systems.

Centralise Log Management

Why : In distributed systems, consolidate logs from multiple sources into a central log management solution.

Benefit : Simplifies monitoring and analysis, enabling more effective incident detection and response. Tools like ELK Stack, Splunk, and Graylog are popular choices.

Secure and Protect Log Data

Why : Logs often contain sensitive information which must be protected.

Benefit : Prevents sensitive data exposure and complies with data protection regulations (like GDPR).

Regularly Monitor and Analyse Logs

Why : Active log monitoring helps identify and respond to issues promptly.

Benefit : Reduces system downtime and can alert to emerging issues before they become critical.

Manage Log Volume

Why : Avoid logging too much unnecessary information.

Benefit : It helps manage log sizes, reduces storage costs, and improves log readability.

Automate Log Analysis

Why : Use tools and scripts to analyse logs for patterns and anomalies automatically.

Benefit : Enhances the ability to quickly identify issues without manually sifting through logs, especially in large-scale systems.

Implement Log Retention Policies

Why : Define how long logs should be retained based on the type of data and compliance requirements.

Benefit : It ensures that logs are available for sufficient time for audits and analysis while avoiding unnecessary storage costs.

Use Asynchronous and Non-blocking Logging

Why : Prevent logging operations from impacting application performance.

Benefit : Minimises the performance overhead of logging activities on the main application processes.

Standardise Logs Across Services

Why : Use a consistent format across different services and parts of your application.

Benefit : It simplifies the analysis and correlation of logs from different sources, which is particularly valuable in microservices architectures.

By integrating these best practices into your development and operational processes, you can maximise the benefits of logging, making it a powerful tool for maintaining and improving your software systems’ performance, reliability, and security.

Tools for Log Management

Effective log management is critical for maintaining system performance, ensuring security, and troubleshooting issues across software applications. Here are some of the most popular tools and platforms that help in aggregating, analysing, and managing logs:

ELK Stack

Components : Elasticsearch, Logstash, and Kibana.

Use Case : ELK Stack is one of the most popular open-source choices for log management. Elasticsearch acts as a search and analytics engine, Logstash is used for log ingestion and processing, and Kibana is used for visualisation and querying.

Benefits : Highly customisable and scalable, capable of handling massive volumes of data.

Splunk

Use Case : Splunk is a powerful commercial solution with a web-based interface that provides extensive capabilities for searching, monitoring, and analysing machine-generated data.

Benefits : Known for its advanced analytics features and extensive out-of-the-box functionalities that can handle complex queries across large datasets.

Graylog

Use Case : Graylog is an open-source log management tool that offers centralised log management, efficient analysis, and a user-friendly dashboard.

Benefits : It’s known for its simplicity and efficiency in storing, searching, and analysing large amounts of data.

Fluentd

Use Case : Fluentd is an open-source data collector for unified logging layers, which allows you to unify data collection and consumption for better use and understanding of data.

Benefits : Fluentd is particularly noted for its flexibility and a wide array of plugins that integrate with many data sources and output formats.

Datadog

Use Case : Datadog provides cloud-scale monitoring that includes the ability to collect, search, and analyse log data, as well as infrastructure and application performance monitoring.

Benefits : It offers real-time logs, sophisticated alerting, and seamless integration with various cloud services.

Loggly

Use Case : Loggly provides cloud-based log management services that can analyse large volumes of data and extract useful information, typically geared towards enterprise-level users.

Benefits : Loggly is easy to set up and integrates well with existing applications, providing robust search capabilities and interactive visualisations.

Papertrail

Use Case : Papertrail offers cloud-based log management that focuses on simplicity and fast setup, providing instant log visibility and analysis.

Benefits : Its simplicity makes it ideal for smaller applications or teams needing quick setup without extensive configuration.

Prometheus and Grafana

Use Case : While Prometheus is primarily used for monitoring and alerting, it can be combined with Grafana for log visualisation. This combo is often used for monitoring and visually analysing metrics.

Benefits : Open-source, decisive for visualising trends and patterns, and highly extensible with Grafana’s advanced dashboard capabilities.

Choosing the Right Tool

When selecting a log management tool, consider factors like:

Volume of Data : The amount of log data you generate.

The complexity of the Environment : Whether you are managing logs from a single application or distributed systems.

Budget : Open-source vs. commercial solutions.

Integration Needs : Compatibility with existing tools and platforms in your ecosystem.

Compliance Requirements : Certain industries might need specific features to comply with legal standards.

Each tool has strengths and is suitable for different organisational needs and sizes. The right choice will depend on your specific requirements, including scalability, ease of use, budget, and the specific features you need.

Common Pitfalls

Logging is a powerful tool in software development, but it can also lead to issues if not appropriately managed. Here are some common pitfalls associated with logging and how to avoid them:

Over-Logging

Problem : Logging too much information can clutter log files, make them challenging to manage, and lead to performance degradation.

Solution : Use appropriate log levels to control the verbosity of the logs. Log only what is necessary for troubleshooting and operational monitoring. Regularly review and adjust what is being logged based on current needs.

Under-Logging

Problem : Insufficient logging can leave you without enough information to diagnose issues, especially in production environments.

Solution : Ensure critical paths and user transactions are well-logged. Log errors effectively, including error handling and catching exceptions. Use structured logging to enhance the quality and usefulness of the logs.

Ignoring Logs

Problem : Not regularly reviewing logs can lead to missed opportunities to detect or fix issues early.

Solution : Implement log monitoring and alerting tools to watch for unusual activity or errors actively. Establish routines for checking logs, especially after deployments or changes.

Not Protecting Log Information

Problem : Logs can contain sensitive information, which, if exposed, can lead to security breaches.

Solution : Secure log files by restricting access and using encryption where necessary. Be mindful of what is logged, especially avoiding logging sensitive user data like passwords or personal information.

Inconsistent Logging

Problem : Inconsistent log formats across different parts of an application can make it difficult to correlate events when analysing logs.

Solution : Standardise log formats across the application. Use structured logging formats like JSON to make logs more uniform and more straightforward to analyse.

Lack of Context in Logs

Problem : In complex or distributed systems, logs with sufficient context can be easier to interpret.

Solution : Logs should include contextual information such as timestamps, user IDs, session IDs, and request IDs to clarify the events being logged.

Blocking or Synchronous Logging

Problem : Synchronous logging can negatively impact application performance, causing delays in user-facing operations.

Solution : Use asynchronous logging mechanisms to minimise the impact on application performance and ensure logging does not block critical application workflows.

Poor Log Management and Retention Practices

Problem : Inadequate log rotation and retention policies can lead to manageable log files and increased storage costs.

Solution : Implement log rotation policies and configure appropriate log retention durations based on business needs and compliance requirements.

Failing to Plan for Log Scalability

Problem : As systems grow, the volume of logs can increase dramatically, leading to scalability issues.

Solution : Plan for log scalability from the start. Consider how logs will be handled, stored, and analysed as data volumes grow.

By being aware of these common pitfalls and actively working to avoid them, you can ensure that your logging practices enhance rather than hinder your application’s development and operation. Effective logging practices lead to more maintainable, reliable, and secure software systems.

]]>JavaSecure Coding Practiceshttps://svenruppert.com/posts/secure-coding-practices-access-control/Fri, 03 May 2024 10:41:23 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/secure-coding-practices-access-control/Access control is a security measure that determines who can access resources or perform actions within a system. It involves defining and enforcing policies restricting unauthorised access while allowing authorised users to perform their intended tasks. Access control mechanisms are commonly used in various domains, including computer systems, buildings, and physical assets.<![CDATA[

Access control is a security measure that determines who can access resources or perform actions within a system. It involves defining and enforcing policies restricting unauthorised access while allowing authorised users to perform their intended tasks. Access control mechanisms are commonly used in various domains, including computer systems, buildings, and physical assets.

1. What are the different Access Control ways in software?
   1. Discretionary Access Control (DAC):
   2. Mandatory Access Control (MAC):
   3. Role-Based Access Control (RBAC):
   4. Attribute-Based Access Control (ABAC):
   5. Rule-Based Access Control (RBAC):
2. What are the secure coding practices in Java for Access Control?
   1. Use the Principle of Least Privilege:
   2. Implement Authentication and Authorization:
   3. Avoid Hardcoding Sensitive Information:
   4. Securely Manage Credentials:
   5. Secure Communication:
   6. Sanitize Inputs:
   7. Implement Access Control Checks:
   8. Secure Error Handling:
   9. Regularly Update Dependencies:
   10. Security Testing:
3. What are the pros and cons of Access Control implemented in Java’s static or dynamic semantics?
   1. Static Semantics:
      1. Pros:
      2. Cons:
   2. Dynamic Semantics:
      1. Pros:
      2. Cons:
4. What are common Design Patterns in Java for implementing Access Control?
   1. Proxy Pattern:
   2. Decorator Pattern:
   3. Chain of Responsibility Pattern:
   4. Strategy Pattern:
   5. Facade Pattern:
   6. Observer Pattern:
   7. Singleton Pattern:
   8. Command Pattern:
5. An example in Java for Access Control using a Proxy
   1. Step 1: Define the Document Interface
   2. Step 2: Implement the Document Class
   3. Step 3: Create the Proxy Class
   4. Step 4: Demonstrate the Proxy in Action

In computer systems, access control typically involves the following components:

Identification : Users must identify themselves to the system by providing a username or other unique identifier.

Authentication : After identification, users must authenticate themselves by providing credentials such as passwords, biometric data, or security tokens.

Authorisation : Once authenticated, users are granted access rights based on their identity and the permissions associated with that identity. Authorisation mechanisms define what resources or actions a user is allowed to access.

Enforcement : The system enforces access control policies to ensure that only authorised users can access resources or perform specific actions. This may involve mechanisms such as encryption, access control lists (ACLs), or role-based access control (RBAC).

Access control can be implemented at various levels, including:

Physical access control refers to restricting access to physical locations such as buildings, rooms, or data centres using mechanisms such as locks, keycards, or biometric scanners.

Logical access control : Managing access to digital resources such as files, databases, or networks through software-based mechanisms such as user accounts, permissions, and encryption.

Access control is fundamental to information security and is essential for protecting sensitive data, preventing unauthorised activities, and ensuring compliance with regulations and organisational policies. It helps organisations manage risks associated with unauthorised access and maintain their resources’ confidentiality, integrity, and availability.

What are the different Access Control ways in software?

In software systems, access control mechanisms are implemented to regulate and manage user access to resources. There are several approaches to access control in software:

Discretionary Access Control (DAC):

DAC allows the owner of a resource to control access to that resource. Owners can grant or revoke access permissions to other users or groups. Access control lists (ACLs) and file permissions are typical implementations of DAC. For example, Unix-like operating systems use file permissions (read, write, execute) to control access to files and directories.

Mandatory Access Control (MAC):

MAC enforces access control based on predefined security policies determined by system administrators or security policies. Users do not have control over access rights. Security labels or security clearances are assigned to resources and users. Access decisions are based on the resource’s sensitivity and the user’s clearance level. SELinux (Security-Enhanced Linux) and TrustedBSD are examples of MAC systems.

Role-Based Access Control (RBAC):

RBAC assigns permissions to roles, and users are assigned to specific roles. Users inherit the permissions associated with their roles. Roles represent job functions or responsibilities within an organisation, and access decisions are based on user roles. RBAC simplifies administration by centralising access control management and reducing the complexity of managing individual user permissions.

Attribute-Based Access Control (ABAC):

ABAC evaluates access decisions based on user, resource, and environment attributes. Attributes can include:

• User attributes (e.g., role, department).
• Resource attributes (e.g., sensitivity, type).
• Environmental attributes (e.g., time, location).

Policies define conditions that must be met for access to be granted based on attribute values. ABAC provides a flexible and dynamic access control model that can adapt to changing organisational requirements and contexts.

Rule-Based Access Control (RBAC):

Rule-based access control defines access rules or policies that specify conditions under which access is granted or denied. Access decisions are based on evaluating these rules against the attributes of the user, resource, and environment. Rules can be defined using logical expressions, conditions, or patterns.

These access control models can be combined or extended to meet the specific security requirements of software systems. The choice of access control mechanism depends on factors such as the system’s nature, the data’s sensitivity, and organisational policies and regulations.

What are the secure coding practices in Java for Access Control?

Secure coding practices in Java for access control help developers ensure that their applications enforce proper access control policies to protect sensitive data and prevent unauthorised access. Here are some best practices:

Use the Principle of Least Privilege:

Grant users and components only the permissions they need to perform their tasks and no more. Limit access to sensitive resources to minimise the potential impact of a security breach.

Implement Authentication and Authorization:

Authenticate users before granting access to protected resources. Use robust authentication mechanisms such as username/password, multi-factor authentication, or OAuth. Implement authorisation mechanisms such as role-based access control (RBAC) or attribute-based access control (ABAC) to enforce access control policies based on user roles, permissions, or attributes.

Avoid Hardcoding Sensitive Information:

Avoid hardcoding sensitive information such as passwords, API keys, or cryptographic keys directly into the source code. Instead, store them securely in configuration files, environment variables, or secure key management systems.

Securely Manage Credentials:

Use secure storage mechanisms such as Java KeyStore (JKS) or Key Management Service (KMS) to store and manage sensitive credentials. Avoid storing plaintext passwords or sensitive information in memory longer than necessary. Use secure storage and encryption techniques when handling sensitive data.

Secure Communication:

Use secure communication protocols such as HTTPS (SSL/TLS) to encrypt data transmitted between clients and servers. Verify server certificates to prevent man-in-the-middle attacks and ensure the integrity and authenticity of communication.

Sanitize Inputs:

Validate and sanitise user inputs to prevent injection attacks such as SQL injection, Cross-Site Scripting (XSS), or Command Injection. Input validation libraries or frameworks such as OWASP ESAPI or Hibernate Validator are used to validate and sanitise user inputs.

Implement Access Control Checks:

Perform access control checks at the entry points of sensitive operations or resources. Use access control mechanisms provided by Java frameworks or libraries such as Spring Security or Apache Shiro to enforce access control policies consistently across the application.

Secure Error Handling:

Implement proper error handling mechanisms to avoid exposing sensitive information in error messages. Use custom error messages or error logging mechanisms to provide informative error messages to developers while hiding sensitive details from end-users.

Regularly Update Dependencies:

Keep your Java libraries and dependencies up-to-date to ensure that you are using the latest security patches and fixes. Monitor security advisories and vulnerability databases for known security issues in third-party libraries and promptly apply patches or updates.

Security Testing:

Perform regular security testing, including penetration testing, code reviews, and security scans, to identify and remediate security vulnerabilities in your Java applications. Use automated security testing tools such as OWASP ZAP, SonarQube, or FindBugs to identify common security issues and vulnerabilities in your code.

By following these secure coding practices, Java developers can strengthen the access control mechanisms in their applications and reduce the risk of security breaches and unauthorised access to sensitive data.

What are the pros and cons of Access Control implemented in Java’s static or dynamic semantics?

Access control can be implemented in both Java’s static and dynamic semantics. Here are the pros and cons of each approach:

Static Semantics:

Pros:

Early Detection of Errors : Static access control mechanisms are enforced at compile-time, allowing for early detection of access control violations. This helps identify potential security vulnerabilities before the code is executed.

Compile-Time Optimization : The compiler can optimise static access control checks, improving the application’s performance by reducing runtime overhead.

Enforcement of Policies : Static access control mechanisms enforce access control policies uniformly across the entire codebase, ensuring consistency and reducing the likelihood of human error.

Cons:

Limited Flexibility : Static access control mechanisms may lack the flexibility to adapt to runtime changes or dynamic conditions. This can be problematic when access control policies must be dynamically modified or customised based on runtime context.

Limited Granularity : Static access control mechanisms may have limitations in providing fine-grained access control, especially in complex or dynamic systems where access privileges must be defined at a more granular level.

Dynamic Semantics:

Pros:

Flexibility : Dynamic access control mechanisms can adapt to changing runtime conditions, allowing for more flexible access control policies that can be adjusted based on dynamic factors such as user roles, permissions, or environmental context.

Fine-Grained Access Control : Dynamic access control mechanisms can provide finer-grained access control, allowing for more precise specification of access privileges based on specific runtime conditions or attributes.

Cons:

Runtime Overhead : Dynamic access control checks incur runtime overhead, as access control decisions are made during program execution. This can potentially impact the application’s performance, especially in performance-sensitive systems.

Late Detection of Errors : Dynamic access control mechanisms may only detect access control violations during runtime, which can delay the detection and resolution of security vulnerabilities. If access control policies are not adequately enforced, this increases the risk of security breaches.

Complexity : Dynamic access control mechanisms can introduce complexity, especially in large or distributed systems, making managing and maintaining access control policies challenging.

In summary, the choice between static and dynamic access control semantics in Java depends on factors such as the specific requirements of the application, the level of flexibility needed, and the trade-offs between performance and security. While static access control provides early detection of errors and compile-time optimisation, dynamic access control offers flexibility and fine-grained control but may incur runtime overhead and complexity.

What are common Design Patterns in Java for implementing Access Control?

Several design patterns can be applied in Java to implement access control effectively. Here are some common design patterns used for access control:

Proxy Pattern:

The Proxy pattern allows for controlling access to objects by acting as intermediaries. Proxy objects can enforce access control rules before allowing the client to access the object. Proxies can restrict access to sensitive resources based on user permissions or roles in access control.

Decorator Pattern:

The Decorator pattern enables dynamic augmentation of object behaviour by wrapping them with additional functionality. In access control, decorators can add security checks or authorisation logic around methods that access sensitive resources.

Chain of Responsibility Pattern:

The Chain of Responsibility pattern creates a chain of handler objects, each responsible for processing a request. The request is passed along the chain until it is handled. In access control, a chain of responsibility can be used to enforce access control checks at various application levels, with each handler responsible for verifying specific access rights.

Strategy Pattern:

The Strategy pattern defines a family of algorithms, encapsulates each one, and makes them interchangeable. Clients can choose the appropriate algorithm at runtime. In access control, the strategy pattern can be used to implement different access control strategies (e.g., RBAC, ABAC) and dynamically allow the application to switch between them.

Facade Pattern:

The Facade pattern provides a unified interface to a set of interfaces in a subsystem, making it easier to use. In access control, a facade can hide the complexity of access control mechanisms by providing a simplified interface for clients to interact with while internally handling the access control logic.

Observer Pattern:

The Observer pattern defines a one-to-many dependency between objects, where the state changes in one object are propagated to its dependents. In access control, observers can notify interested parties (e.g., audit logs, monitoring systems) about access control events, such as access granted or denied.

Singleton Pattern:

The Singleton pattern ensures that a class has only one instance and provides a global access point to that instance. In access control, a singleton can be used to manage a centralised access control policy or configuration, ensuring consistency across the application.

Command Pattern:

The Command pattern encapsulates a request as an object, thereby allowing for parameterisation of clients with queues, requests, and operations. In access control, commands can represent access control requests, encapsulating the necessary information (e.g., user, resource, action) to perform access control checks.

By applying these design patterns, developers can design flexible, maintainable, and extensible access control mechanisms in Java applications, ensuring that access to sensitive resources is adequately managed and enforced.

An example in Java for Access Control using a Proxy

To illustrate access control using a proxy in Java, let’s create a simple example where we have aDocument class that users can read or write. We will then use aDocumentProxy class to control access to theDocument based on the user’s role (e.g., “admin” can read and write, while “user” can only read).

Step 1: Define the Document Interface

This interface declares the methods for interacting with a document.

javaCopy

publicinterfaceDocument{voiddisplayDocument();voidmodifyDocument(Stringcontent);}

Step 2: Implement the Document Class

This class implements theDocument interface. It represents a real document that can be modified and displayed.

javaCopy

publicclassRealDocumentimplementsDocument{privateStringcontent;publicRealDocument(Stringcontent){this.content=content;}@OverridepublicvoiddisplayDocument(){System.out.println("Displaying Document: "+content);}@OverridepublicvoidmodifyDocument(Stringcontent){this.content=content;System.out.println("Document Modified");}}

Step 3: Create the Proxy Class

TheDocumentProxy class controls access to theRealDocument based on the user’s role.

javaCopy

publicclassDocumentProxyimplementsDocument{privateRealDocumentrealDocument;privateStringuserRole;publicDocumentProxy(RealDocumentrealDocument,StringuserRole){this.realDocument=realDocument;this.userRole=userRole;}@OverridepublicvoiddisplayDocument(){realDocument.displayDocument();}@OverridepublicvoidmodifyDocument(Stringcontent){if("admin".equals(userRole)){realDocument.modifyDocument(content);}else{System.out.println("Access Denied: You do not have permission to modify the document.");}}}

Step 4: Demonstrate the Proxy in Action

Finally, let’s create a simple demo to illustrate how the proxy controls access based on the user role.

javaCopy

publicclassProxyDemo{publicstaticvoidmain(String[]args){RealDocumentdocument=newRealDocument("Original Content");//user trying to modify the documentDocumentProxyuserProxy=newDocumentProxy(document,"user");userProxy.displayDocument();userProxy.modifyDocument("New User Content");// Admin trying to modify the documentDocumentProxyadminProxy=newDocumentProxy(document,"admin");adminProxy.displayDocument();adminProxy.modifyDocument("New Admin Content");adminProxy.displayDocument();}}

In this example, when a “user " tries to modify the document through the “DocumentProxy “, access is denied. However, when an “admin " attempts to change it, the operation is successful. This demonstrates how a proxy can be used to control access to certain functionalities based on user roles.

]]>JavaSecure Coding Practiceshttps://svenruppert.com/posts/the-hidden-dangers-of-bidirectional-characters/Fri, 19 Apr 2024 10:12:58 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/the-hidden-dangers-of-bidirectional-characters/Discover the hidden dangers of bidirectional control characters! We dive deep into how these essential text-rendering tools can be exploited to manipulate digital environments. Learn about their security risks, from filename spoofing to deceptive URLs, and uncover the crucial strategies to safeguard against these subtle yet potent threats. Understand how to protect your systems in a multilingual world. Join to ensure your digital security is not left to chance!<![CDATA[

Discover the hidden dangers of bidirectional control characters! We dive deep into how these essential text-rendering tools can be exploited to manipulate digital environments. Learn about their security risks, from filename spoofing to deceptive URLs, and uncover the crucial strategies to safeguard against these subtle yet potent threats. Understand how to protect your systems in a multilingual world. Join to ensure your digital security is not left to chance!

1. Key Bidirectional Control Characters
   1. Left-to-Right Mark (LRM) - U+200E:
   2. Right-to-Left Mark (RLM) - U+200F:
   3. Left-to-Right Embedding (LRE) - U+202A:
   4. Right-to-Left Embedding (RLE) - U+202B:
   5. Pop Directional Formatting (PDF) - U+202C:
   6. Left-to-Right Override (LRO) - U+202D:
   7. Right-to-Left Override (RLO) - U+202E:
2. Uses and Applications
3. Some Demos
   1. Java Demo - Right-to-Left Override (RLO) Attack
   2. Explanation
   3. Java Demo - Right-to-Left Mark (RLM) Attack
   4. Explanation
4. But why is this a security issue?
   1. File Name Spoofing
   2. Phishing Attacks
   3. Code Obfuscation
   4. Misleading Data and Database Entries
   5. User Interface Deception
5. Addressing the Security Risks
   1. Input Validation and Sanitization
   2. Secure Default Configurations
   3. User and Administrator Education
   4. Enhanced Monitoring and Logging
   5. Security Policies and Procedures
   6. Technological Solutions
6. Conclusion:

Key Bidirectional Control Characters

Bidirectional control characters (often abbreviated as bidi control characters) are special characters used in text encoding to manage the direction of text flow. This is crucial for languages read right-to-left (RTL), like Arabic and Hebrew, when mixed with left-to-right (LTR) languages like English. These characters help to ensure that the text is displayed in the correct order, regardless of the directionality of its parts.

Here are some of the common bidi control characters defined in the Unicode standard:

Left-to-Right Mark (LRM) - U+200E:

They are used to set the direction of the text from left to right. It is particularly useful when embedding a small piece of LTR text within a larger segment of RTL text.

Right-to-Left Mark (RLM) - U+200F:

This setting sets the direction of the text to right-to-left. It is used when embedding a small RTL text within a larger segment of LTR text.

Left-to-Right Embedding (LRE) - U+202A:

They are used to start a segment of LTR text within an RTL environment. This embedding level pushes onto the directional status stack.

Right-to-Left Embedding (RLE) - U+202B:

They are used to start a segment of RTL text within an LTR environment.

Pop Directional Formatting (PDF) - U+202C:

They are used to end a segment of embedded text, popping the last direction from the stack and returning to the previous directional context.

Left-to-Right Override (LRO) - U+202D:

It forces the text within its scope to be treated as left-to-right text, regardless of its directionality. This is useful for reordering sequences of characters.

Right-to-Left Override (RLO) - U+202E:

Forces the text within its scope to be treated as right-to-left text, even if it is typically LTR. This can be used to display text backwards, which might be used for effect or in specific contexts.

Uses and Applications

Bidirectional control characters are essential for the following:

Multilingual Documents : Ensuring coherent text flow when documents contain multiple languages with different reading directions.

User Interfaces : Proper text rendering in software that supports multiple languages.

Data Files : Manage data display in multiple languages with different directionalities.

Some Demos

Bidirectional control characters can pose security risks. They can be used to obscure the true intent of a code or text, leading to what is known as a “bidirectional text attack.” For instance, filenames could appear to end with a harmless extension like “.txt” when they end with a dangerous one like “.exe” reversed by bidi characters. As a result, users might need to be more informed about the nature of the files they interact with.

Security-aware text editors and systems often have measures to detect and appropriately display or alert users about the presence of bidirectional control characters to mitigate potential security risks.

Here’s a simple Java demo that illustrates how bidirectional control characters can be used to create misleading filenames. This can demonstrate the potential danger, particularly in environments where filenames are manipulated or displayed based on user input.

Java Demo - Right-to-Left Override (RLO) Attack

This demo will:

Create a seemingly harmless text file named “txt.exe” using bidirectional control characters. The file will output the actual and displayed names to show the discrepancy.

javaCopy

importjava.io.File;importjava.io.IOException;publicclassBidiDemo{publicstaticvoidmain(String[]args){// U+202E is the Right-to-Left Override (RLO) characterStringnormalName="report.txt";StringdeceptiveName="report"+"\u202E"+"exe.txt";// Try to create files with these namescreateFile(normalName);createFile(deceptiveName);// Print what the names look like to the Java programSystem.out.println("Expected file name: "+normalName);System.out.println("Deceptive file name appears as: "+deceptiveName);}privatestaticvoidcreateFile(StringfileName){Filefile=newFile(fileName);try{if(file.createNewFile()){System.out.println("File created: "+file.getName());}else{System.out.println("File already exists: "+file.getName());}}catch(IOExceptione){System.out.println("An error occurred while creating the file: "+fileName);e.printStackTrace();}}}

Explanation

Creation of Names : The deceptive file name is created using the right-to-left override character (U+202E). This causes the part of the filename after the bidi character to be interpreted as right-to-left, making “exe.txt” look like “txt.exe” in some file systems and interfaces.

File Creation : The program attempts to create files with standard and deceptive names.

Output Differences : When printed, the deceptive name will show the filename reversed after the bidi character, potentially misleading users about the file type and intent.

To see the effect:

- Compile and run the Java program.

- Check the output and the file system to observe how the filenames are displayed.

Java Demo - Right-to-Left Mark (RLM) Attack

Let’s examine a Java example that demonstrates how a Right-to-Left Mark (RLM) can be critical in ensuring the correct display and handling of mixed-direction text. This example will simulate a simple scenario where Arabic and English text are combined, highlighting how the RLM character helps maintain the intended order of words.

This Java example will:

1. Combine English and Arabic text in a single string.

2. Use the Right-to-Left Mark (RLM) to manage the display order correctly.

3. Print out the results to illustrate the effect of using RLM.

javaCopy

publicclassRLMExample{publicstaticvoidmain(String[]args){// Arabic reads right to left, English left to rightStringenglishText="Version 1.0";StringarabicText="الإصدار";// Concatenate without RLMStringwithoutRLM=arabicText+" "+englishText;// Concatenate with RLMStringwithRLM=arabicText+"\u200F"+" "+englishText;// Print the resultsSystem.out.println("Without RLM: "+withoutRLM);System.out.println("With RLM: "+withRLM);}}

Explanation

Arabic and English Text : Arabic is inherently right-to-left, whereas English is left-to-right.

Concatenation without RLM : Depending on the environment, simply concatenating Arabic and English text might not always display correctly, as the directionality of the English text can disrupt the flow of the Arabic.

Concatenation with RLM : By inserting a Right-to-Left Mark after the Arabic text but before the English text, the English part is correctly treated as part of the right-to-left sequence. This ensures the English text is read in its natural order but positioned correctly within the overall RTL context.

When you run this program, especially in a console or environment that supports bidirectional text:

The “Without RLM” output may show the English text misplaced or improperly aligned relative to the Arabic text.

The “With RLM” output should show the English text correctly placed and maintain the natural reading order of both languages.

This example underscores the importance of RLM in software and user interfaces dealing with multilingual data. It ensures that text is presented in a way that respects the reading order of different languages. Proper handling of bidirectional text is crucial in applications ranging from document editors to web content management systems.

But why is this a security issue?

Bidirectional control characters like the Right-to-Left Mark (RLM) are a security concern primarily due to their ability to obscure the true intent of text and data. This ability can be exploited in various ways to mislead users or automated systems about the content or function of data, leading to potential security vulnerabilities. Here are some specific scenarios where this becomes critical:

File Name Spoofing

One of the most common security issues related to bidirectional control characters is file name spoofing. Attackers can use bidi characters to reverse the order of characters in a file’s extension in file names, making a malicious executable file appear as a harmless type, such as a text file. For instance, the file nameddoc.exe might be displayed asexe.cod in systems that do not handle bidi characters properly, tricking users into thinking it’s merely a document.

Phishing Attacks

In phishing emails or misleading links, bidi characters can be used to reverse parts of a URL to mimic a trusted domain, leading users to malicious sites. For example, what appears to beexample.com in reversed parts could be a link to an entirely different and dangerous site, exploiting the user’s trust in familiar-looking URLs.

Code Obfuscation

Developers or malicious coders might use bidi characters to obscure code logic or comments in software, making it difficult for security analysts or automated tools to assess the code’s behaviour accurately. This can hide malicious functions or bypass security audits.

Misleading Data and Database Entries

Bidi characters can be used to reverse strings in database entries, potentially leading to incorrect or misleading data processing. This could be exploited to bypass filters and validation checks or to intentionally corrupt data integrity.

User Interface Deception

In applications with user interfaces that display user input data, bidi characters can create a misleading representation of that data. This could need to be clarified for users or lead them to make incorrect decisions based on incorrectly displayed information.

Addressing the Security Risks

Addressing the security risks associated with bidirectional control characters (bidi characters) requires a multifaceted approach that includes technical safeguards and user education. Here are more detailed strategies that organizations and software developers can employ to mitigate these risks:

Input Validation and Sanitization

Strict Validation Rules : Implement strict validation rules that check for the presence of bidi characters in sensitive contexts such as file names, URLs, and input forms. This validation should identify and flag or reject unexpected or unauthorized use of these characters.

Character Filtering : For applications not requiring bidi characters, remove them from inputs during the data entry or ingestion process. For applications where such characters are necessary, ensure they are used correctly and safely.

Encoding Techniques : Use encoding techniques to handle potentially dangerous characters safely. For example, HTML entities can encode bidi characters in web applications, preventing them from being processed as active components of the code.

Secure Default Configurations

Display Controls : Configure systems and applications to visually distinguish or neutralize bidi characters, particularly in environments where their use is rare or unexpected. This could involve displaying their unicode point instead of the character or providing visual indicators of text direction changes.

Limit Usage Contexts : Restrict the contexts in which bidi characters can be used, especially in identifiers like usernames, filenames, and URLs, unless there is a specific need for them.

User and Administrator Education

Awareness Training : Conduct regular training sessions for users and administrators about potentially misusing bidi characters and other Unicode anomalies. Include real-world examples of how these features can be exploited.

Best Practices for Content Creation : Educate content creators on the correct and safe use of bidi characters, emphasizing the security aspects of text directionality in content that will be widely distributed or used in sensitive environments.

Enhanced Monitoring and Logging

Anomaly Detection : Use advanced monitoring tools to detect unusual bidi character usage patterns in system logs, network traffic, or transaction data. This can help identify potential attacks or breaches early.

Audit Trails : Maintain robust audit trails, including detailed logging of input validation failures and other security-related events. This can help with forensic analysis and understanding attack vectors after a security incident.

Security Policies and Procedures

Clear Policies : Develop and enforce clear security policies regarding handling bidi characters. This includes guidelines for developers handling text input and output and policies for content managers reviewing and approving content.

Incident Response : Include the misuse of bidi characters as a potential vector in your organization’s incident response plan. Prepare specific procedures to respond to incidents involving deceptive text or file manipulations.

Technological Solutions

Development Frameworks and Libraries : Utilize frameworks and libraries that inherently handle bidi characters safely and transparently. Ensure that these tools are up-to-date and configured correctly.

User Interface Design : Design user interfaces that inherently mitigate the risks posed by bidi characters, such as displaying full file extensions and using text elements that visually separate user input from system text.

Implementing these strategies requires a coordinated effort between software developers, security professionals, system administrators, and end-users. Organizations can significantly reduce the risks of bidi characters and other related security threats by adopting comprehensive measures.

Conclusion:

In conclusion, while often overlooked, the security risks associated with bidirectional control characters are significant and can have profound implications for individuals and organizations. These characters can be exploited in various deceptive ways, from file name spoofing and phishing attacks to code obfuscation and misleading data presentations. To effectively mitigate these risks, a comprehensive and multi-layered approach is necessary.

This approach should include stringent input validation and sanitization processes to filter out or safely handle bidi characters where they are not needed and to ensure they are used appropriately where they are necessary. Secure default configurations that visually indicate the presence and effect of bidi characters can help prevent their misuse, while robust monitoring and logging can aid in detecting and responding to potential security threats.

Education also plays a crucial role. Users and administrators need to be aware of how bidi characters can be used maliciously, and developers need to be informed about best practices for handling such characters in their code. Security policies must be clear and enforced, with specific guidelines on handling bidi characters effectively and safely.

Finally, employing technological solutions that can handle these characters appropriately and designing user interfaces that mitigate their risks will further strengthen an organization’s defence against the security vulnerabilities introduced by bidirectional control characters. By addressing these issues proactively, we can safeguard the integrity of digital environments and protect sensitive information from being compromised.

]]>JavaSecurityhttps://svenruppert.com/posts/audio-steganography-in-more-detail/Wed, 17 Apr 2024 19:22:20 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/audio-steganography-in-more-detail/Audio steganography is a technique for hiding information within an audio file so that only the intended recipient knows of the hidden data’s existence. This method belongs to the broader field of steganography, which itself is a subset of security systems and comes from the Greek words “steganos,” meaning covered, and “graphein,” meaning writing.<![CDATA[

Audio steganography is a technique for hiding information within an audio file so that only the intended recipient knows of the hidden data’s existence. This method belongs to the broader field of steganography, which itself is a subset of security systems and comes from the Greek words “steganos,” meaning covered, and “graphein,” meaning writing.

The primary objective of audio steganography is to conceal the presence of secret data by embedding it into an audio signal without noticeable degradation of the signal. Unlike cryptography, which secures the contents of a message through encryption, steganography focuses on concealing the fact that there is a hidden message.

Methods of Audio Steganography

There are various techniques used to embed data within audio files, including:

Least Significant Bit (LSB) Insertion:

This is one of the simplest methods where bits of the hidden data are inserted into the least significant bits of the audio file. Because these modifications are slight, they are generally imperceptible to the human ear.

Phase Coding:

Phase coding is a sophisticated technique used in audio steganography to embed information within an audio file by manipulating the phase of the sound signal. This method leverages the fact that the human auditory system is far less sensitive to phase changes than it is to changes in amplitude or frequency. As such, phase coding can be a very effective way to hide information without noticeable changes to the sound quality as perceived by human listeners.

How Phase Coding Works

Phase coding is typically applied to the phase spectrum of a sound signal. The process involves several steps:

Signal Decomposition: The original audio signal is divided into segments using a Fast Fourier Transform (FFT) transform. This decomposition converts the time-domain signal into the frequency domain, where each component has an amplitude and a phase.

Phase Manipulation: The phase of the audio file’s initial segment (or a reference segment) is adjusted to embed the secret data. The phases of subsequent segments are then altered to ensure a smooth transition between segments, maintaining the perceptual integrity of the audio signal. This is crucial to prevent artefacts that could be detectable by the human ear.

Data Embedding: The binary data to be hidden is typically encoded in the phase changes between successive segments. A simple method might involve using the presence or absence of a phase shift as a binary one or zero respectively.

Signal Reconstruction: After the phase modification, the segments are transformed back to the time domain, recombining them to produce the final audio signal with the hidden data embedded.

Advantages of Phase Coding

Subtlety: Since human ears are not particularly sensitive to phase variations, especially in complex sounds with many overlapping frequencies, this method can hide data effectively without audible distortion.

Robustness to Compression: Phase coding can be relatively robust against lossy compression, especially compared to methods like LSB insertion, which such processes can disrupt.

Challenges and Considerations

Complexity: Implementing phase coding requires careful handling to maintain the coherence of the phase between segments. Poor implementation can result in noticeable audio artifacts.

Data Capacity: While phase coding is discreet, it typically offers lower data capacity compared to other steganographic techniques. This is because excessive manipulation of phase information can lead to perceptible sound quality degradation or inconsistencies.

Dependency on Sound Content: The effectiveness of phase coding can depend heavily on the type of audio being used. Complex signals with many frequency components (like music) offer more opportunities for phase manipulation without detection than more straightforward signals (like speech).

In summary, phase coding is a powerful but complex technique in audio steganography, offering a high level of discretion for embedding data within audio files. Its successful application requires careful handling to balance data capacity, sound quality, and robustness against signal processing.

Spread Spectrum:

In this method, the secret message is spread across the frequency spectrum of the audio file. This spreading makes the message less susceptible to intentional or unintentional modifications.

This technique is based on spread spectrum communication technology principles, initially developed for military use to ensure secure and robust communication over radio waves.

How Spread Spectrum Works

In the context of audio steganography, spread spectrum involves embedding a secret message into an audio signal by slightly altering its frequency components. The process typically follows these steps:

Data Preparation: The data to be hidden is first prepared, often by encoding it in a binary format. This binary data is then spread out over a larger bandwidth than it would normally occupy, which helps in disguising its presence.

Signal Spreading: The spread data is combined with a pseudo-noise code (PN code) or a similar key that only the sender and intended recipient know. The PN code has properties similar to noise but with a known structure, making it possible to detect and decode the data without being detected by unintended listeners.

Modulation: The combined data and PN code modulate the host audio signal, typically using techniques like Direct Sequence Spread Spectrum (DSSS) or Frequency Hopping Spread Spectrum (FHSS). In DSSS, the signal is spread across a wide range of frequencies based on the PN code. In FHSS, the signal frequency hops rapidly in a pattern defined by the PN code.

Integration into Audio Signal: The modulated signal is then subtly integrated into the audio file. This integration is done so that the modifications to the audio are imperceptible to the human ear but can be detected and decoded by analyzing the audio spectrum with the correct key.

Advantages of Spread Spectrum

Robustness: Spread spectrum techniques are highly resistant to interference and noise. Because the data is spread across a wide band of frequencies, it remains intact even if parts of the signal are disrupted or lost.

Security: The use of a PN code makes the hidden data difficult to detect and decode without the correct key, providing an additional layer of security.

Low Detectability: The hidden data’s wide distribution across the frequency spectrum and low amplitude make it difficult to detect through casual listening or even with spectral analysis without prior knowledge.

Challenges and Considerations

Complexity: Implementing spread spectrum in audio steganography requires sophisticated signal processing techniques and careful tuning to ensure that the hidden data does not affect the quality of the audio.

Bandwidth Requirements: The method requires more bandwidth to spread the data, which can be a limitation in environments where bandwidth is constrained.

Dependency on Audio Content: Like other steganographic methods, the effectiveness and capacity of the spread spectrum can depend on the nature of the audio content. Complex audio signals with a wide dynamic range and rich frequency content are better for hiding data.

Echo Hiding:

This involves introducing an echo into the discrete signal. Parameters like the amplitude, decay rate, and offset of the echo can be manipulated to embed data. This method capitalizes on the auditory characteristics of human perception, specifically how humans perceive echoes, to hide data effectively without noticeably altering the quality of the audio.

How Echo Hiding Works

Echo hiding works by manipulating the properties of echo—such as delay, decay, and amplitude—to encode data. The basic steps involved in echo hiding are:

Echo Creation: Echoes are artificially created and superimposed onto the original audio signal. The parameters of these echoes—such as the delay (time between the original sound and its echo) and the decay rate (how quickly the echo fades away)—are crucial.

Data Embedding: Binary data is encoded into the audio signal by varying the parameters of the echoes. For instance, a short delay might represent a binary ‘0’ while a longer delay might represent a binary ‘1’. The amplitude of the echo can also be used to encode data, with different levels of loudness representing different data bits.

Parameter Control: To remain imperceptible, the echoes must be subtle. The echo delay is typically kept within the range of 1 to 3 milliseconds, as delays shorter than 1 millisecond are generally not perceived as echoes, and those longer than 3 milliseconds can begin to be perceived as discrete repeats rather than reverberation or natural echo.

Signal Synthesis: After encoding the data, the modified signal (original plus echo) is synthesized back into a coherent audio stream. This new audio stream contains the hidden information encoded within the echo parameters but should sound nearly identical to the original to an unsuspecting listener.

Advantages of Echo Hiding

Imperceptibility: Since the echoes are subtle and use natural auditory phenomena, the modifications are typically imperceptible to casual listeners.

Robustness: Echo properties can be robust against certain types of signal processing, such as compression and transmission over noisy channels because the characteristics of the echo (especially if embedded in a robust part of the audio spectrum) can remain detectable even if the quality of the audio is somewhat degraded.

Compatibility: Echo hiding does not require significant alteration of the frequency content of the audio signal, which helps preserve the original quality and characteristics of the audio.

Challenges and Considerations

Detection and Removal: While robust against some forms of manipulation, sophisticated audio analysis tools designed to identify and modify echo characteristics can detect and potentially remove echoes.

Capacity Limitations: Compared to other steganographic methods, echo hiding generally hides a limited amount of data. Overloading the signal with too many echoes can make it more detectable and degrade the audio quality.

Dependency on Content: The effectiveness of echo hiding can depend on the nature of the audio content. Audio signals with lots of natural variation and existing reverberation may mask the steganographic echoes better than very dry, sparse, or highly dynamic signals.

Frequency Masking:

Frequency masking is an audio steganography technique that exploits the limitations of human auditory perception to hide information within an audio file. It leverages a phenomenon known as “auditory masking,” where certain sounds become inaudible in the presence of other, louder sounds at similar frequencies. This technique is particularly subtle because it embeds data in a way that is naturally concealed by the characteristics of the audio itself.

How Frequency Masking Works

The basic concept of frequency masking involves embedding hidden data into parts of the audio spectrum where the presence of louder, dominant sounds will mask it. The process typically involves the following steps:

Analysis of Audio Spectrum: The first step is to analyze the frequency spectrum of the host audio signal to identify potential masking opportunities. This involves finding frequency ranges where louder sounds are likely to mask quieter ones.

Data Preparation: The data intended for hiding is prepared, usually encoded into a binary format. This data needs to be modulated or otherwise processed to fit within the selected masked frequencies.

Embedding Data: The prepared data is then embedded into the quieter parts of the audio spectrum, specifically within the critical bands of frequency where masking is most effective. Critical bands are frequency ranges in which the human ear processes sound as a single auditory event.

Signal Synthesis: After embedding the data, the audio signal is reconstructed to include the hidden data. This process must be handled delicately to ensure that the modifications do not become perceptible, maintaining the quality and integrity of the original audio.

Advantages of Frequency Masking

Imperceptibility: Because the hidden data is placed in regions where louder sounds naturally mask it, it is very difficult to detect without specific, sophisticated analysis tools.

Robustness to Compression: Frequency masking can be relatively robust against some forms of audio compression, particularly if the embedded data is strategically placed in less compressible parts of the spectrum.

Utilization of Auditory Phenomena: This method uses a natural auditory phenomenon, making it a very organic form of steganography.

Challenges and Considerations

Complex Signal Analysis Required: Effective use of frequency masking requires detailed analysis of the audio signal’s spectral properties, which can be complex and computationally intensive.

Limited Data Capacity: The amount of data that can be hidden is generally limited to the available masked regions, which may not be extensive depending on the audio content.

Dependency on Audio Content: The success of frequency masking heavily depends on the nature of the audio file. Audio tracks with dense and varied spectral content provide more opportunities for masking than simpler, cleaner tracks.

Applications

Audio steganography has various applications across multiple fields. Some of these include:

Secure Communications: These are used by organizations and individuals to communicate sensitive information discreetly.

Watermarking: Audio files can be watermarked to assert ownership, much like watermarking images or videos.

Covert Operations: Used by law enforcement and military for operations requiring secure and stealthy communication methods.

Challenges

Despite its advantages, audio steganography faces several challenges:

Robustness:

The steganographic information must remain intact even if the audio file undergoes compression, format conversion, or other types of digital processing.

Imperceptibility:

The alterations made to embed the data should not be detectable by normal hearing, as this would compromise the steganographic integrity.

Capacity:

The amount of data that can be hidden is generally limited by the size of the host file and the technique used, which could be restrictive for larger data needs.

Conclusion

Audio steganography offers a unique way to conceal information within audio files, making it a valuable tool for security and privacy in digital communications. Its effectiveness lies in its ability to hide information in plain sight, providing an added layer of security through obscurity. However, its success and reliability depend heavily on the choice of technique and the nature of the application. As technology evolves, so do steganography methods, which are continuously improving to meet the demands of modern security needs.

]]>Securityhttps://svenruppert.com/posts/beyond-the-visible-exploring-the-depths-of-steganography/Thu, 28 Mar 2024 14:02:52 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/beyond-the-visible-exploring-the-depths-of-steganography/Steganography is the practice of concealing a message, file, image, or video within another message, file, image, or video. Unlike cryptography, which focuses on making a message unreadable to unauthorised parties, steganography aims to hide the message’s existence. The word “steganography " is derived from the Greek words “steganos ,” meaning “covered ,” and “graphein ,” meaning “to write.”<![CDATA[

Steganography is the practice of concealing a message, file, image, or video within another message, file, image, or video. Unlike cryptography, which focuses on making a message unreadable to unauthorised parties, steganography aims to hide the message’s existence. The word “steganography " is derived from the Greek words “steganos ,” meaning “covered ,” and “graphein ,” meaning “to write.”

1. An example from history
2. Example of using Steganography in Cybersecurity
3. How to do it practically in Java
   1. Text Steganography:
   2. Image Steganography:
   3. Audio Steganography:
   4. Video Steganography:
   5. File Steganography:
4. Conclusion:

An example from history

One notable historical example of steganography involves using invisible ink during the American Revolutionary War. The British and American forces employed various forms of steganography to conceal messages and strategic information.

In particular, the Culper Spy Ring, a clandestine network of American spies operating during the Revolutionary War, extensively used invisible ink to communicate covertly. One member of the ring, invisible ink expert James Jay, developed a secret method for creating invisible ink using simple household ingredients such as lemon juice or milk.

The Culper Spy Ring would write messages with this invisible ink between the lines of innocent-looking letters or on the blank spaces of documents. To reveal the hidden messages, the recipient would apply heat or special chemicals, causing the invisible ink to darken and become visible.

This use of steganography allowed the Culper Spy Ring to transmit crucial intelligence about British troop movements, plans, and other sensitive information without detection by British authorities. The effective use of invisible ink by the Culper Spy Ring played a significant role in aiding the American cause during the Revolutionary War and exemplifies the historical importance of steganography in espionage and covert operations.

https://youtu.be/tYeC3A57wgc

Example of using Steganography in Cybersecurity

A modern example of steganography in cybersecurity involves concealing malicious code within seemingly innocuous digital files to bypass security measures and deliver malware to targeted systems. Cybercriminals often employ this technique to evade detection by traditional antivirus software and intrusion detection systems.

For instance, attackers may embed malicious payloads, such as Trojans or ransomware, within images, audio files, or documents using steganography techniques. The carrier files appear unchanged to the naked eye or standard file analysis tools, making it difficult for security solutions to detect hidden malware.

Once the steganographically encoded file reaches the target system, the attacker can extract and execute the concealed payload, thereby compromising the system and initiating malicious activities.

To combat this threat, cybersecurity professionals utilise advanced threat detection technologies capable of analysing files for signs of steganographic manipulation. These tools employ various techniques, such as statistical analysis, anomaly detection, and signature-based detection, to identify suspicious patterns or deviations from expected file structures.

Additionally, cybersecurity awareness training programs educate users about the risks associated with opening files from untrusted sources and emphasise the importance of maintaining up-to-date security software to mitigate the threat of steganography-based attacks.

All Code Examples are on GitHub:https://github.com/svenruppert/Steganography

How to do it practically in Java

Steganography techniques can involve various methods, such as:

The source code you will find on GitHub under the following URL:

https://github.com/svenruppert/Steganography

Text Steganography:

Embedding secret messages within text documents by altering spacing, font styles, or using invisible characters.

Here’s a simple example of text steganography in Java using a basic technique called whitespace steganography. In this example, we’ll hide a secret message within a text document by manipulating the whitespace between words.

javaCopy

publicclassTextSteganographyExample{45publicstaticvoidmain(String[]args){6StringbinaryData="101";// Binary data to hide7StringsourceText="Hello\nWorld\nThis is a test\n";// Source text to hide data in8StringhiddenText=hideData(sourceText,binaryData);910System.out.println("Text without hidden data:");11System.out.println(sourceText);12System.out.println("Text with hidden data:");13System.out.println(hiddenText);1415//extract the hidden message16StringextractedData=extractData(hiddenText);17System.out.println("extractedData = "+extractedData);18}1920publicstaticStringhideData(StringsourceText,StringbinaryData){21Scannerscanner=newScanner(sourceText);22StringBuilderhiddenTextBuilder=newStringBuilder();23intindex=0;24while(scanner.hasNextLine()&&index<binaryData.length()){25Stringline=scanner.nextLine();26// Append a space for '0', or a tab for '1'27charappendChar=binaryData.charAt(index)=='0'?' ':'\t';28hiddenTextBuilder.append(line).append(appendChar).append("\n");29index++;30}31// If there's more of the source text, add it as is32while(scanner.hasNextLine()){33hiddenTextBuilder.append(scanner.nextLine()).append("\n");34}35scanner.close();36returnhiddenTextBuilder.toString();37}3839publicstaticStringextractData(StringhiddenText){40Scannerscanner=newScanner(hiddenText);41StringBuilderbinaryDataBuilder=newStringBuilder();42while(scanner.hasNextLine()){43Stringline=scanner.nextLine();44if(line.endsWith("\t")){45// If line ends with a tab, append '1'46binaryDataBuilder.append('1');47}elseif(line.endsWith(" ")){48// If line ends with a space, append '0'49binaryDataBuilder.append('0');50}51}52scanner.close();53returnbinaryDataBuilder.toString();54}55}

This code demonstrates a basic form of text steganography where a secret message is hidden within the whitespace of a text document. Note that this method needs to be revised and may not be suitable for high-security applications. Advanced techniques can involve more sophisticated manipulation of text or embedding data within specific patterns.

Image Steganography:

Data is concealed within digital images by modifying the least significant bits of pixel values or by embedding data within specific regions of the image where slight alterations are less noticeable.

Certainly! Here’s a simple example of image steganography in Java using LSB (Least Significant Bit) embedding. In this example, we’ll hide a secret message within the least significant bits of an image’s pixels.

This code hides a secret message within the least significant bit of each pixel in the image. The original image is saved as a new steganographic image when the message is hidden. Later, the hidden message is extracted by reading the steganographic image’s pixels and retrieving the least significant bit of the red component of each pixel. Finally, the binary message is converted back into a readable string. Replace “input_image.png” with the path to your input image.

javaCopy

publicclassImageSteganography{89// Method to encode a message into an image10publicstaticBufferedImageencodeMessage(BufferedImageimage,Stringmessage){11intmessageLength=message.length();12intimageWidth=image.getWidth();13intimageHeight=image.getHeight();14int[]imagePixels=newint[imageWidth*imageHeight];15image.getRGB(0,0,imageWidth,imageHeight,imagePixels,0,imageWidth);1617// Convert the message into a binary string18StringBuilderbinaryMessage=newStringBuilder();19for(charcharacter:message.toCharArray()){20binaryMessage.append(String.format("%8s",Integer.toBinaryString(character)).replaceAll(" ","0"));21}2223// Encode the message length at the beginning24StringmessageLengthBinary=String.format("%32s",Integer.toBinaryString(messageLength)).replace(' ','0');25binaryMessage.insert(0,messageLengthBinary);2627// Encode the binary message into the image28for(inti=0;i<binaryMessage.length();i++){29intpixel=imagePixels[i];30intblue=pixel&0xFF;31intgreen=(pixel>>8)&0xFF;32intred=(pixel>>16)&0xFF;33intalpha=(pixel>>24)&0xFF;3435// Modify the LSB of the blue part of the pixel to match the current bit of the message36blue=(blue&0xFE)|(binaryMessage.charAt(i)-'0');37intnewPixel=(alpha<<24)|(red<<16)|(green<<8)|blue;3839imagePixels[i]=newPixel;40}4142BufferedImagenewImage=newBufferedImage(imageWidth,imageHeight,BufferedImage.TYPE_INT_ARGB);43newImage.setRGB(0,0,imageWidth,imageHeight,imagePixels,0,imageWidth);44returnnewImage;45}46// Method to decode the hidden message from an image47publicstaticStringdecodeMessage(BufferedImageimage){48intimageWidth=image.getWidth();49intimageHeight=image.getHeight();50int[]imagePixels=newint[imageWidth*imageHeight];51image.getRGB(0,0,imageWidth,imageHeight,imagePixels,0,imageWidth);5253// Extract the length of the message54StringBuildermessageLengthBinary=newStringBuilder();55for(inti=0;i<32;i++){56intpixel=imagePixels[i];57intblue=pixel&0xFF;58messageLengthBinary.append(blue&1);59}60intmessageLength=Integer.parseInt(messageLengthBinary.toString(),2);6162// Extract the binary message from the image63StringBuilderbinaryMessage=newStringBuilder();64for(inti=32;i<32+messageLength*8;i++){65intpixel=imagePixels[i];66intblue=pixel&0xFF;67binaryMessage.append(blue&1);68}6970// Convert the binary message to string71StringBuildermessage=newStringBuilder();72for(inti=0;i<binaryMessage.length();i+=8){73StringbyteString=binaryMessage.substring(i,i+8);74intcharCode=Integer.parseInt(byteString,2);75message.append((char)charCode);76}7778returnmessage.toString();79}80publicstaticvoidmain(String[]args)throwsIOException{81FileoriginalImageFile=newFile("_data/_DSC1259.png");// Specify the path to the input image82BufferedImageoriginalImage=ImageIO.read(originalImageFile);83StringsecretMessage="Secret message goes here";8485// Encode the message into the image86BufferedImageencodedImage=encodeMessage(originalImage,secretMessage);8788// Save the encoded image89FileoutputImageFile=newFile("_data/_DSC1259_with-data.png");// Specify the path to the output image90ImageIO.write(encodedImage,"png",outputImageFile);91System.out.println("The message has been encoded into the image.");929394FileimageFile=newFile("_data/_DSC1259_with-data.png");// Specify the path to the encoded image95BufferedImageimage=ImageIO.read(imageFile);9697// Decode the message from the image98StringdecodedMessage=decodeMessage(image);99System.out.println("The hidden message is: "+decodedMessage);100calculatingMaxInfoAmount("_data/_DSC1259_with-data.png");101}102privatestaticvoidcalculatingMaxInfoAmount(Stringpathname){103try{104FileimageFile=newFile(pathname);// Specify the path to your image105BufferedImageimage=ImageIO.read(imageFile);106intimageWidth=image.getWidth();107intimageHeight=image.getHeight();108longmaxBits=(long)imageWidth*imageHeight;// Maximum bits that can be stored109longmaxBytes=maxBits/8;// Convert bits to bytes110111System.out.println("Maximum information that can be stored in bytes: "+maxBytes);112}catch(IOExceptione){113thrownewRuntimeException(e);114}115}116}

Audio Steganography:

Hiding information within audio files by modifying the least significant bits of audio samples or by exploiting imperceptible frequencies.

Here’s a basic example of audio steganography in Java using LSB (Least Significant Bit) embedding. In this example, we’ll hide a secret message within the least significant bits of the audio samples.

javaCopy

publicclassAudioSteganography{67publicstaticvoidmain(String[]args){8FileinputFile=newFile("_data/Security - 2024.06 - What is Steganography-16bit-single-track_A01_L.wav");9FileoutputFile=newFile("_data/output.wav");10Stringmessage="Secret Message";11hideMessage(inputFile,outputFile,message);1213FileinputFileEncoded=newFile("_data/output.wav");// This is the file with the hidden message14StringextractedMessage=extractMessage(inputFileEncoded,14);// Assuming we know the message length15System.out.println("Extracted Message: "+extractedMessage);16}1718publicstaticvoidhideMessage(FileinputFile,FileoutputFile,Stringmessage){19try{20// Convert the message to a binary string21byte[]messageBytes=message.getBytes();22StringBuilderbinaryMessage=newStringBuilder();23for(byteb:messageBytes){24binaryMessage.append(String.format("%8s",Integer.toBinaryString(b&0xFF)).replace(' ','0'));25}26StringmessageBinary=binaryMessage.toString();2728// Load the audio file29AudioInputStreamaudioInputStream=AudioSystem.getAudioInputStream(inputFile);30AudioFormatformat=audioInputStream.getFormat();31byte[]audioBytes=audioInputStream.readAllBytes();3233// Hide the message in the LSB of the audio bytes34intmessageIndex=0;35for(inti=0;i<audioBytes.length&&messageIndex<messageBinary.length();i+=2){// Skip every other byte for 16-bit samples36if(messageBinary.charAt(messageIndex)=='1'){37audioBytes[i]=(byte)(audioBytes[i]|1);// Set LSB to 138}else{39audioBytes[i]=(byte)(audioBytes[i]&~1);// Set LSB to 040}41messageIndex++;42}4344// Write the modified samples to a new file45ByteArrayInputStreambais=newByteArrayInputStream(audioBytes);46AudioInputStreamoutputAudioInputStream=newAudioInputStream(bais,format,audioBytes.length/format.getFrameSize());47AudioSystem.write(outputAudioInputStream,AudioFileFormat.Type.WAVE,outputFile);4849System.out.println("The message has been hidden in "+outputFile.getName());5051}catch(UnsupportedAudioFileException|IOExceptione){52e.printStackTrace();53}54}5556publicstaticStringextractMessage(FileinputFile,intmessageLength){57try{58// Load the audio file59AudioInputStreamaudioInputStream=AudioSystem.getAudioInputStream(inputFile);60byte[]audioBytes=audioInputStream.readAllBytes();6162// Extract bits to reconstruct the message binary string63StringBuildermessageBinary=newStringBuilder();64for(inti=0;i<messageLength*8*2;i+=2){// Assuming 16-bit samples, adjust for actual sample size65byteb=audioBytes[i];66intlsb=b&1;// Extract the LSB67messageBinary.append(lsb);68}6970// Convert the binary string to text71StringBuildermessage=newStringBuilder();72for(inti=0;i<messageBinary.length();i+=8){73StringbyteString=messageBinary.substring(i,i+8);74intcharCode=Integer.parseInt(byteString,2);75message.append((char)charCode);76}7778returnmessage.toString();7980}catch(UnsupportedAudioFileException|IOExceptione){81e.printStackTrace();82}8384returnnull;85}86}

This code hides a secret message within the input audio file’s least significant bit of each audio sample. The modified audio data is then saved as a new steganographic audio file. Later, the hidden message is extracted by reading the least significant bit of each audio sample in the steganographic audio file. Replace filename and path with the path to your input audio file.

Video Steganography:

Concealing data within video files by manipulating frames or embedding data in specific segments.

Below is a basic example of video steganography in Java using LSB (Least Significant Bit) embedding. In this example, we’ll hide a secret message within the least significant bits of the blue pixel values of video frames.

javaCopy

publicclassVideoSteganography{14//Should be extracted from the orig video15publicstaticfinalintFPS=50;16publicstaticfinalStringINPUT_FILE="input.mp4";17publicstaticfinalStringOUTPUT_FILE="output.mp4";1819publicstaticvoidmain(String[]args)throwsException{20FileChannelWrapperfileChannelWrapperIN=NIOUtils.readableChannel(newFile(INPUT_FILE));21FileoutputFile=newFile(OUTPUT_FILE);22// Create a SequenceEncoder for the output file at 25 frames per second23SequenceEncoderencoder=SequenceEncoder.createSequenceEncoder(outputFile,FPS);24FrameGrabgrab=FrameGrab.createFrameGrab(fileChannelWrapperIN);25Picturepicture;26// to improve to process27// 0. detect the FPS from the input video28// 1. calculate the max amount of possible bytes that can be encoded29// 2. calculate the max amount of bytes that can be stored in each frame30// 3. split the message into chunks to fit into a frame31// 4. define a sequence to mark the end of the message3233while(null!=(picture=grab.getNativeFrame())){34// Here, convert the Picture to BufferedImage35BufferedImageframe=AWTUtil.toBufferedImage(picture);36// Modify the frame to encode part of the message37BufferedImageencodedMessageOnBlue=encodeMessageOnBlue(frame,toBinaryString("Hello Message"));38// Convert BufferedImage to Picture (required by JCodec)39Picturepic=AWTUtil.fromBufferedImage(encodedMessageOnBlue,ColorSpace.RGB);40// Encode the modified frame back into the video41encoder.encodeNativeFrame(pic);42}43// Finalize video encoding and clean up44NIOUtils.closeQuietly(fileChannelWrapperIN);45// Finalize and close the encoder (important!)46encoder.finish();47}4849privatestaticStringtoBinaryString(Stringmessage){50StringBuilderbinary=newStringBuilder();51for(charcharacter:message.toCharArray()){52binary.append(String53.format("%8s",Integer.toBinaryString(character))54.replace(' ','0'));55}56returnbinary.toString();57}5859privatestaticBufferedImageencodeMessageOnBlue(BufferedImageimage,StringbinaryMessage){60intmessageIndex=0;61intmessageLength=binaryMessage.length();6263for(inty=0;y<image.getHeight()&&messageIndex<messageLength;y++){64for(intx=0;x<image.getWidth()&&messageIndex<messageLength;x++){65intpixel=image.getRGB(x,y);66intalpha=(pixel>>24)&0xFF;67intred=(pixel>>16)&0xFF;68intgreen=(pixel>>8)&0xFF;69intblue=pixel&0xFF;7071// Modify the LSB of the blue component72blue=(blue&0xFE)|(binaryMessage.charAt(messageIndex)-'0');73messageIndex++;7475// Reconstruct the pixel and set it back76intnewPixel=(alpha<<24)|(red<<16)|(green<<8)|blue;77image.setRGB(x,y,newPixel);78}79}80returnimage;81}82}

File Steganography:

Embedding files within other files, such as hiding a document within another document.

File steganography with PDF files involves hiding one PDF file within another PDF file. In this example, we’ll hide the contents of one PDF file within the metadata of another PDF file. Specifically, we’ll utilise the “Keywords " metadata field of PDFs to embed the content of a secret PDF file.

Here’s the Java code to achieve this using the Apache PDFBox library:

javaCopy

910publicclassHideMessageInPDF{11publicstaticvoidmain(String[]args){12try{13// Load an existing PDF document14Filefile=newFile("_data/ReadME.pdf");15PDDocumentdocument=PDDocument.load(file);16// Retrieve the document's metadata17PDDocumentInformationinfo=document.getDocumentInformation();18// Add a hidden message to the metadata19// You could use a less obvious key than "HiddenMessage" to make it less detectable20info.setCustomMetadataValue("HiddenMessage","This is a secret message");21// Save the modified document22document.save("_data/ReadME_with-data.pdf");23document.close();24System.out.println("Hidden message added to the PDF metadata.");25}catch(IOExceptione){26e.printStackTrace();27}2829try{30// Load the PDF document31Filefile=newFile("_data/ReadME_with-data.pdf");32PDDocumentdocument=PDDocument.load(file);33// Access the document's metadata34PDDocumentInformationinfo=document.getDocumentInformation();35// Retrieve the hidden message from the metadata36StringhiddenMessage=info.getCustomMetadataValue("HiddenMessage");37System.out.println("Hidden message: "+hiddenMessage);38document.close();39}catch(IOExceptione){40e.printStackTrace();41}42}43}

In this code:

1. The “hidePDF " function loads the source PDF document, converts the content of the secret PDF file to a Base64 encoded string, and embeds it into the “Keywords " metadata field of the source PDF file.

2. The “extractPDF " function loads the steganographic PDF document, extracts the Base64 encoded content from the “Keywords " metadata field, decodes it, and writes it to an output PDF file.

Make sure to replace every path to files with the appropriate file paths in your system. Additionally, you’ll need to include the Apache PDFBox library in your project to use its functionalities.

Conclusion:

As explored in this paper, steganography is a fascinating field with a rich history and diverse applications. By concealing information from ordinary data, steganography provides a powerful means of covert communication and data protection. Steganography continues to evolve alongside technological advancements, from ancient techniques of hiding messages in wax tablets to modern digital methods embedded within multimedia files.

While steganography offers numerous benefits in fields like digital watermarking, copyright protection, and secure communication, it also presents challenges and ethical considerations. Due to its subtle nature, detecting steganographic content remains a significant challenge, requiring advanced algorithms and tools for effective analysis.

As technology continues to advance, the importance of understanding steganography becomes increasingly critical. It underscores the need for robust security measures balanced with respect for privacy rights. By delving into the principles and techniques of steganography, researchers and practitioners can develop more effective strategies for exploiting and defending against covert communication methods.

In conclusion, steganography represents a dynamic and intriguing aspect of information security, with implications spanning various domains. By embracing its complexities and exploring its potential applications, we can navigate the intricate landscape of digital communication with greater insight and resilience.

]]>JavaSecurityhttps://svenruppert.com/posts/the-compensating-transaction-pattern/Mon, 12 Feb 2024 12:40:41 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/the-compensating-transaction-pattern/The Bird-Eye View A Compensating Transaction Pattern is a technique used to ensure consistency when multiple steps are involved in a process, and some steps may fail. It essentially consists in having “undo” transactions for each successful step, so if something goes wrong later on, you can reverse the changes made earlier and maintain data integrity.<![CDATA[

The Bird-Eye View

A Compensating Transaction Pattern is a technique used to ensure consistency when multiple steps are involved in a process, and some steps may fail. It essentially consists in having “undo” transactions for each successful step, so if something goes wrong later on, you can reverse the changes made earlier and maintain data integrity.

Here’s a breakdown of the key points:

What it does:

• Ensures consistency in eventually consistent operations with multiple steps.
• Reverses the work done by previous successful steps if a later step fails.
• Maintains data integrity by undoing changes in case of failure.

How it works:

• Primary transaction: A series of steps are performed to complete a specific task.
• Compensating transactions: A “undo” transaction is designed and stored for each successful step.
• Failure: If a later step fails, the compensating transactions for the completed steps are executed reversely, effectively undoing the changes.

1. The Bird-Eye View
   1. What it does:
   2. How it works:
   3. Benefits:
   4. Examples of use cases:
      1. Online Hotel Booking System:
      2. Inventory Management System:
      3. Money Transfer Service:
      4. Flight Reservation System:
      5. Order Processing System - (e-commerce):
2. Tradeoffs:
   1. Complexity:
   2. Concurrency and Race Conditions:
   3. Performance Overhead:
      1. Overhead of Compensating Actions:
      2. Latency and Response Time:
      3. Concurrency and Scalability:
      4. Network and I/O Overhead:
      5. Transaction Retry and Rollback:
      6. Resource Consumption:
      7. Monitoring and Logging Overhead:
   4. Data Consistency:
   5. Error Handling and Recovery:
   6. Auditability and Monitoring:
   7. Development and Testing Complexity:
      1. Compensating Action Design:
      2. Testing Failure Scenarios:
      3. Concurrency and Race Conditions:
      4. Error Handling and Recovery Testing:
      5. Integration Testing:
      6. End-to-End Testing:
3. Unsuccessful Transaction Example: Online Shopping Fail
   1. Main Transaction:
   2. Compensating Transactions:
   3. Failure:
   4. Compensating Actions:
   5. Outcome:
   6. Additional Notes:
4. Implementing Compensating Transactions in Core Java (Simplified Example)
5. What other Patterns are often combined?
   1. Retry Pattern:
   2. Saga Pattern:
   3. Event Sourcing Pattern:
   4. Circuit Breaker Pattern:
   5. Idempotent Receiver Pattern:
6. Conclusion:

Benefits:

The Compensating Transaction Pattern offers several benefits in the context of distributed systems and complex transactions:

Fault Tolerance : The pattern enhances fault tolerance by providing a mechanism to handle failures gracefully. When a failure occurs during a transaction, compensating actions can be executed to revert the system to a consistent state, ensuring that partial failures do not leave the system in an inconsistent or corrupted state.

Consistency : Compensating transactions helps maintain data consistency across distributed systems by ensuring that changes made by failed transactions are properly reverted. This helps prevent data anomalies and ensures the system remains consistent, even during failures.

Transaction Rollback : Unlike traditional transaction rollback mechanisms, compensating transactions offer more flexibility in handling complex transactions that involve multiple components or services. Instead of rolling back the entire transaction, compensating actions can selectively undo the effects of failed transactions, allowing other parts of the transaction to proceed unaffected.

Resilience : By incorporating compensating transactions into the system design, developers can improve the resilience of distributed systems against various types of failures, including network issues, service unavailability, or system crashes. This resilience contributes to overall system stability and reliability.

Flexibility : The pattern provides flexibility in designing and implementing transactional workflows in distributed systems. Developers can define compensating actions tailored to specific transactional requirements, allowing for custom handling of failures and recovery scenarios.

Enhanced Scalability : Compensating transactions can facilitate the implementation of scalable distributed systems by decoupling transactional logic from individual components or services. This decoupling allows for better scalability and parallelisation of transaction processing across distributed environments.

Improved Error Handling : Compensating transactions enables more robust error handling mechanisms by providing a structured approach to handling transaction failures. Developers can implement specific error-handling logic within compensating actions to address various failure scenarios, such as resource constraints or service timeouts.

Overall, the Compensating Transaction Pattern offers significant benefits in fault tolerance, consistency, resilience, flexibility, scalability, and error handling, making it a valuable pattern for designing and implementing distributed systems with complex transactional requirements.

Examples of use cases:

Here are five examples of the Compensating Transaction Pattern in action across various domains:

Online Hotel Booking System:

Main Transaction : Reserve a hotel room for a customer and charge their credit card.

Compensating Actions : If charging the credit card fails or the reservation cannot be made due to a system error, release the reserved room and void the transaction authorisation on the credit card.

Inventory Management System:

Main Transaction : Deduct inventory stock when an order is placed.

Compensating Actions : If an order cannot be fulfilled due to insufficient stock or system errors, increment the stock count for the ordered items to revert the deduction.

Money Transfer Service:

Main Transaction : Transfer funds from one account to another.

Compensating Actions : If the transfer fails due to a network issue or insufficient funds, reverse the transfer by depositing the amount back into the sender’s account.

Flight Reservation System:

Main Transaction : Book a flight ticket for a passenger and debit the payment from their account.

Compensating Actions : If the payment fails or the reservation cannot be completed, cancel the booking and refund the payment to the passenger’s account.

Order Processing System - (e-commerce):

Main Transaction : Process an order by updating inventory, charging the customer, and notifying the warehouse for shipment.

Compensating Actions : If any step fails (e.g., payment authorisation fails, inventory update fails, or shipment notification fails), revert the changes made and inform the customer about the failure.

In each of these examples, the Compensating Transaction Pattern ensures that the system maintains consistency and integrity, even in the face of failures or errors during transaction processing. Compensating actions are designed to undo the effects of failed transactions and restore the system to a consistent state.

Tradeoffs:

While the Compensating Transaction Pattern offers several benefits, it also comes with inevitable tradeoffs and considerations that developers need to take into account:

Complexity:

Implementing compensating transactions can add complexity to the system design and codebase. Developers must carefully design and manage compensating actions to ensure they accurately revert the effects of failed transactions. This complexity can make the codebase harder to understand, maintain, and debug.

Concurrency and Race Conditions:

In distributed systems with concurrent transactions, coordinating compensating actions across multiple components or services can introduce race conditions and synchronisation challenges. Ensuring that compensating actions are executed correctly and idempotently in the presence of concurrent transactions requires careful attention to concurrency control mechanisms.

Performance Overhead:

Executing compensating actions to revert failed transactions incurs additional processing overhead and resource utilisation. Depending on the complexity of compensating actions and the frequency of transaction failures, this overhead can impact system performance and scalability, particularly in high-throughput environments.

The performance implications of the Compensating Transaction Pattern depend on various factors, including the complexity of the transactional workflow, the frequency of transaction failures, the efficiency of compensating actions, and the underlying infrastructure of the distributed system. Here are some performance considerations associated with this pattern:

Overhead of Compensating Actions:

Executing compensating actions to revert failed transactions incurs additional processing overhead and resource utilisation. Depending on the complexity of compensating actions, their execution may involve database updates, service calls, or other operations that consume CPU, memory, and network bandwidth.

Latency and Response Time:

The time required to execute compensating actions impacts the overall latency and response time of transaction processing. If compensating actions involve time-consuming operations or depend on external services, they can introduce delays in recovering from transaction failures, affecting system responsiveness and user experience.

Concurrency and Scalability:

Coordinating compensating actions across concurrent transactions can introduce contention and synchronisation overhead, impacting system scalability. In highly concurrent environments, ensuring that compensating actions are executed correctly and efficiently without causing bottlenecks or resource contention is crucial for maintaining performance under load.

Network and I/O Overhead:

Compensating actions involving remote services or accessing distributed data sources incur network and I/O overhead. Network latency, message serialisation/deserialisation, and network congestion can affect the performance of compensating transactions, particularly in geographically distributed systems.

Transaction Retry and Rollback:

Implementing retry mechanisms for failed transactions and rolling back partially completed transactions involve additional computational and I/O overhead. Retrying failed transactions increases the workload on the system, primarily if retries are performed with exponential backoff or involve multiple retries before triggering compensating actions.

Resource Consumption:

Compensating transactions may consume additional resources such as memory, disk space, and database connections. Managing resource usage and contention, especially in multi-tenant or shared infrastructure environments, is essential for maintaining overall system performance and stability.

Monitoring and Logging Overhead:

Capturing transactional events, logging execution outcomes, and monitoring the performance of compensating transactions impose overhead on system resources, particularly in environments with high transaction volumes. Balancing the granularity of monitoring and logging with performance considerations is essential for optimising system performance.

To mitigate the performance implications of the Compensating Transaction Pattern, developers can employ various strategies such as optimising compensating actions, reducing transactional complexity, implementing efficient retry mechanisms, optimising network communication, scaling infrastructure resources, and using caching and batching techniques. Additionally, conducting performance testing and profiling to identify and address performance bottlenecks is essential for optimising the performance of systems that utilise compensating transactions.

Data Consistency:

While compensating transactions helps maintain consistency in the face of transaction failures, ensuring data consistency across distributed systems remains a challenge. Coordinating compensating actions with other concurrent transactions and ensuring that data updates are properly synchronised can be complex, especially in systems with eventual consistency requirements.

Here’s an example illustrating the potential data consistency problem:

Consider an e-commerce platform where customers can place orders for products. When a customer places an order, the system deducts the ordered items from the inventory and charges the customer’s credit card. If either of these actions fails, compensating actions are executed to revert the transaction.

Let’s suppose a customer orders a product, the inventory is successfully deducted, but the credit card charge fails due to a network issue. The compensating action would be adding the deducted items to the inventory. However, due to the system’s distributed nature, there’s a delay in executing the compensating action; in the meantime, another customer orders the same product.

If the compensating action to add back the deducted items is not synchronised or appropriately coordinated, it may lead to data inconsistency. Specifically, the inventory count may need to be corrected if the compensating action adds back the deducted items while another order for the same product is already processed. This can result in overselling products and accurate inventory counts, leading to customer satisfaction and operational inefficiencies.

To address this data consistency problem, developers need to implement proper synchronisation and coordination mechanisms to ensure that compensating actions are executed in a consistent and idempotent manner. This may involve using distributed transactions, locks, or other coordination techniques to coordinate compensating actions across distributed components and maintain data consistency. Additionally, implementing mechanisms to handle concurrent updates and conflicts, such as optimistic concurrency control or conflict resolution strategies, can help mitigate data consistency issues using the Compensating Transaction Pattern.

Error Handling and Recovery:

Implementing robust error handling and recovery mechanisms for compensating transactions is essential but challenging. Developers need to anticipate and handle various failure scenarios, including partial failures, network issues, and service unavailability, to ensure that compensating actions are executed reliably and effectively.

Auditability and Monitoring:

Tracking and auditing compensating transactions for accountability and compliance purposes can be challenging, especially in complex distributed systems with multiple transactional workflows. Implementing comprehensive monitoring and logging mechanisms to capture transactional events and execution outcomes is crucial for troubleshooting and compliance.

Development and Testing Complexity:

Developing and testing compensating transactions require thoroughly validating the main transactional workflow and compensating actions. Testing failure scenarios and edge cases to ensure the correctness and robustness of compensating transactions can be time-consuming and resource-intensive.

Here’s an example illustrating the complexities involved:

Consider a scenario where you’re developing an online banking system that allows users to transfer funds between accounts. The system must deduct funds from the sender’s account and credit them to the recipient’s account. If the transfer fails (e.g., insufficient funds, network error), compensating actions must be executed to revert the transaction and restore the accounts to their original states.

Now, during development and testing of this system, you encounter several challenges related to the Compensating Transaction Pattern:

Compensating Action Design:

Designing compensating actions requires careful consideration of the actions needed to revert the effects of failed transactions. In the banking system example, you must define compensating actions to restore the sender’s account balance if the transfer fails. This may involve adding back the deducted funds and updating transaction logs or audit trails.

Testing Failure Scenarios:

Testing the system under various failure scenarios to ensure that compensating actions are triggered and executed correctly adds complexity to the testing process. For example, you must simulate scenarios such as insufficient funds, network timeouts, and service failures to verify that compensating actions are invoked and revert the transaction as expected.

Concurrency and Race Conditions:

Testing concurrent transactions and ensuring that compensating actions are executed correctly in a concurrent environment requires thorough testing and validation. Race conditions and synchronisation issues may arise when multiple transactions are processed concurrently, leading to inconsistent or incorrect compensating action execution.

Error Handling and Recovery Testing:

Validating error handling and recovery mechanisms to ensure that the system gracefully handles failures and recovers consistently adds to the testing complexity. This includes testing retry mechanisms, rollback procedures, and error recovery paths to verify that the system behaves as expected under failure conditions.

Integration Testing:

Testing the integration of compensating transactions with other system components and services introduces additional complexity. To maintain data consistency and integrity, you need to ensure that compensating actions interact correctly with external dependencies, such as databases, messaging systems, and third-party services.

End-to-End Testing:

To validate the entire transactional workflow, including main transactions and compensating actions, end-to-end testing requires comprehensive testing scenarios and test coverage. This involves testing different transaction sequences, error scenarios, and edge cases to verify the correctness and robustness of the system.

Overall, addressing the development and testing complexity associated with the Compensating Transaction Pattern requires thorough planning, design, and validation to ensure that compensating actions are correctly implemented, tested, and integrated into the system. Effective testing strategies, automation, and collaboration between development and testing teams are essential for mitigating the challenges and complexities of implementing this pattern.

In summary, while the Compensating Transaction Pattern offers benefits such as fault tolerance, consistency, and resilience, it also introduces complexity, concurrency challenges, performance overhead, and additional data consistency, error handling, auditability, and testing considerations. Understanding these tradeoffs and carefully designing compensating transactions is essential for effectively leveraging this pattern in distributed systems.

Unsuccessful Transaction Example: Online Shopping Fail

Imagine you’re buying a new pair of shoes online. Here’s an example of how a Compensating Transaction Pattern could be used in this scenario, and how it would work if the transaction failed:

Steps:

Main Transaction:

• You add the shoes to your cart and proceed to checkout.
• You enter your billing and shipping information.
• You authorise the payment for the shoes.
• The store’s inventory system updates, marking the shoes as “sold.”
• The order confirmation email is sent to you.

Compensating Transactions:

For each successful step:

• A compensating transaction is designed and stored.
• For example, if needed, the inventory update would have a compensating transaction to reverse the “sold” status.

Failure:

Let’s say the shipping address verification fails after successful payment authorisation. The main transaction cannot be completed.

Compensating Actions:

• The compensating transaction for the inventory update is triggered.
• The shoes are marked as “available” again.
• The payment authorisation is reversed, and the funds are returned to your account.
• You receive a notification about the failed transaction and the reason for failure.

Outcome:

• While the purchase wasn’t successful, the Compensating Transaction Pattern ensures data consistency.
• Your money is safe, and the store’s inventory remains accurate.

Additional Notes:

• This is a simplified example, and real-world implementations can be more complex.
• The specific compensating transactions depend on the particular systems and processes involved.

Implementing Compensating Transactions in Core Java (Simplified Example)

Here’s a simplified example in Core Java showcasing the essential aspects of the Compensating Transaction Pattern:

Scenario : Booking a hotel room with payment.

Classes :

• HotelBookingService: Maintains hotel booking logic.
• PaymentService: Handles payment processing.
• Booking: Represents a hotel room booking with details.

Code:

javaCopy

publicclassHotelBookingService{privatefinalPaymentServicepaymentService;publicHotelBookingService(PaymentServicepaymentService){this.paymentService=paymentService;}publicBookingbookRoom(StringguestName,StringroomType,doubleprice){// 1. Book the room (potentially in a database)Bookingbooking=newBooking(guestName,roomType,price);// ... (simulate database booking)// 2. Process paymentbooleanpaymentSuccess=paymentService.charge(price);if(paymentSuccess){returnbooking;// success}else{// 3. Compensate if payment fails:cancelBooking(booking);thrownewRuntimeException("Payment failed, booking cancelled.");}}privatevoidcancelBooking(Bookingbooking){// ... (simulated database cancellation)System.out.println("Booking cancelled for room: "+booking.getRoomType());}}publicclassPaymentService{publicbooleancharge(doubleamount){// ... (simulated payment processing - can throw exceptions)System.out.println("Processing payment of $"+amount);// For simplicity, always succeed in this examplereturntrue;}}publicclassBooking{privateStringguestName;privateStringroomType;privatedoubleprice;// getters, setters, and constructors omitted for brevity}

Explanation :

• “HotelBookingService.bookRoom ” initiates the main transaction:

javaCopy

*Bookstheroom(simulatedwithacomment).*ProcessespaymentusingPaymentService.

• If payment succeeds, the booked room is returned.
• If paymentfails :

javaCopy

*"**cancelBooking** "iscalledtoundotheroombooking(simulated).*A"**RuntimeException** "isthrowntoindicatefailure.

Note :

• This is a simplified example for learning purposes and doesn’t handle real-world complexities like concurrency, resource management, error handling, and persistence.
• Payment service is always successful here for simplicity. In reality, it might fail, requiring more elaborate compensation logic.

This is just a single example to illustrate the concepts. Actual implementations vary based on specific scenarios and technology stacks.

What other Patterns are often combined?

The Compensating Transaction Pattern is often combined with other design patterns to create robust and flexible solutions for handling complex transactions and distributed systems. Some patterns that are frequently combined with the Compensating Transaction Pattern include:

Retry Pattern:

This pattern involves retrying an failed operation, hoping it will succeed on subsequent attempts. Combining this pattern with compensating transactions can improve fault tolerance by automatically retrying the failed operation before executing compensating actions.

Saga Pattern:

The Saga Pattern decomposes a long-running transaction into a series of smaller, localised transactions. Each step in the saga is executed atomically, and compensating transactions are defined to undo the effects of completed steps if a failure occurs. This pattern is highly compatible with compensating transactions and helps manage complex distributed transactions.

Event Sourcing Pattern:

In event sourcing, the state of an application is determined by a sequence of events. This pattern can be combined with compensating transactions to ensure that events representing successful transactions are persisted only after the compensating actions for the entire transaction have been successfully executed.

Circuit Breaker Pattern:

The Circuit Breaker Pattern detects and prevents repeated failures when calling remote services. When combined with compensating transactions, the circuit breaker can temporarily stop sending requests to a failing service, allowing the system to execute compensating actions and recover from the failure.

Idempotent Receiver Pattern:

This pattern ensures receiving systems can safely process the same message multiple times without causing unintended side effects. When combined with compensating transactions, idempotency guarantees that executing compensating actions numerous times does not lead to inconsistencies in the system.

By combining these patterns, developers can design resilient and scalable distributed systems that can effectively handle failures, maintain consistency, and recover from errors gracefully.

Conclusion:

In conclusion, the Compensating Transaction Pattern offers valuable benefits such as fault tolerance, data consistency, and resilience in distributed systems by providing a mechanism to handle transaction failures and maintain system integrity. However, leveraging this pattern effectively requires careful consideration of its tradeoffs and complexities.

While compensating transactions enhance fault tolerance and data consistency, they introduce development and testing complexity due to the intricacies of designing and validating compensating actions and ensuring the correctness and robustness of transactional workflows. Challenges such as concurrency, error handling, integration, and end-to-end testing require thorough planning, design, and validation to mitigate.

Despite these challenges, the Compensating Transaction Pattern remains a powerful tool for building resilient and scalable distributed systems. By addressing the complexities and tradeoffs associated with this pattern through effective design, testing, and validation strategies, developers can harness its benefits to build reliable and fault-tolerant systems that meet the demands of modern distributed computing environments.

]]>Design PatternJavahttps://svenruppert.com/posts/serialising-in-java-birds-eye-view/Sun, 11 Feb 2024 13:46:53 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/serialising-in-java-birds-eye-view/Serialisation in Java is implemented to convert the state of an object into a byte stream, which can be quickly persisted to a file or sent over a network. This process is essential for persisting object data, supporting network communication, and facilitating sharing of objects between different parts of a distributed system.<![CDATA[

Serialisation in Java is implemented to convert the state of an object into a byte stream, which can be quickly persisted to a file or sent over a network. This process is essential for persisting object data, supporting network communication, and facilitating sharing of objects between different parts of a distributed system.

Basics of Serialization:

Object Serialization:

- Serialization is converting an object into a sequence of bytes.

- The serialised form of an object can be saved to a file or sent over a network.

- Deserialization is the process of reconstructing the object from its serialised form.

Serializable Interface:

- In Java, thejava.io.Serializable interface makes a class serialisable.

- This interface acts as a marker interface, indicating that class objects can be serialised.

Transient Keyword:

- Thetransient keyword can mark instance variables that should not be serialised.

- Transient variables are not included in the serialised form when an object is serialised.

javaCopy

classMyClassimplementsSerializable{intnormalVar;transientinttransientVar;}

The Serialisation Process:

ObjectOutput/InputStream:

- TheObjectOutputStream class is used for serialising objects.

- TheObjectInputStream class is used for deserialising objects.

Serialisable Fields:

- Only the serialisation process will include the fields of a class marked as serialisable (i.e., non-transient and their types are serialisable).

Serialisable Methods:

- If a class provides its ownwriteObject andreadObject methods, these methods will be responsible for the custom serialisation and deserialisation logic.

javaCopy

privatevoidwriteObject(ObjectOutputStreamout)throwsIOException{// Custom serialisation logic}privatevoidreadObject(ObjectInputStreamin)throwsIOException,ClassNotFoundException{// Custom deserialisation logic}

Serialisation Control:

SerialVersionUID:

- Every serialisable class has a version number, known asserialVersionUID.

- It is used during deserialisation to verify that the sender and receiver of a serialised object have loaded classes for that object that are compatible concerning serialisation.

Externalisable Interface:

In Java, the**Externalizable** interface provides a way to customise an object’s serialisation and deserialisation process. Unlike theSerializable interface, which performs automatic serialisation,**Externalizable** allows you more control over the process by implementing two methods:**writeExternal** and**readExternal**.

Here’s a simple example to demonstrate the use ofExternalizable:

javaCopy

importjava.io.*;classPersonimplementsExternalizable{privateStringname;privateintage;// Required public no-argument constructorpublicPerson(){}publicPerson(Stringname,intage){this.name=name;this.age=age;}@OverridepublicvoidwriteExternal(ObjectOutputout)throwsIOException{// Custom serialization logicout.writeObject(name);out.writeInt(age);}@OverridepublicvoidreadExternal(ObjectInputin)throwsIOException,ClassNotFoundException{// Custom deserialization logicname=(String)in.readObject();age=in.readInt();}@OverridepublicStringtoString(){return"Person{name='"+name+"', age="+age+"}";}}publicclassExternalizableExample{publicstaticvoidmain(String[]args){// Create a Person objectPersonperson=newPerson("John",30);// Serialize the object to a filetry(ObjectOutputStreamoos=newObjectOutputStream(newFileOutputStream("person.ser"))){oos.writeObject(person);}catch(IOExceptione){e.printStackTrace();}// Deserialize the object from the filePersondeserializedPerson=null;try(ObjectInputStreamois=newObjectInputStream(newFileInputStream("person.ser"))){deserializedPerson=(Person)ois.readObject();}catch(IOException|ClassNotFoundExceptione){e.printStackTrace();}// Print the deserialized objectif(deserializedPerson!=null){System.out.println("Deserialized Person: "+deserializedPerson);}}}

In this example, thePerson class implements theExternalizable interface and provides custom serialisation and deserialisation logic in the**writeExternal** and**readExternal** methods. Themain method demonstrates how to create aPerson object, serialise it to a file, and then deserialise it back into a new object.

Why should I use Externalizable?

The**Externalizable** interface in Java provides a more flexible and customised approach to serialisation compared to the default serialisation mechanism provided by theSerializable interface. Here are some reasons you might want to useExternalizable:

Selective Serialization :

With**Externalizable**, you have complete control over which fields get serialised and how they are serialised. This can be useful when you exclude certain fields from serialisation or perform custom serialisation for specific fields.

Performance Optimization:

**Externalizable** allows you to implement custom serialisation logic that might be more efficient than the default serialisation mechanism. You can choose to serialise only essential data, avoiding unnecessary information and potentially reducing the size of the serialised data.

Versioning Control:

When your class evolves and you need to maintain backwards or forward compatibility,Externalizable provides a way to handle versioning explicitly. You can implement custom logic to manage different versions of the serialised data, making it more adaptable to changes in your class structure.

Security Considerations:

Custom serialisation logic can help handle security-related concerns. For example, encrypt sensitive information before serialisation or perform other security checks during the serialisation/deserialisation process.

Resource Management:

With**Externalizable**, you control resource management during serialisation and deserialisation. You can explicitly release and acquire resources as needed, which can be crucial when resources must be managed carefully.

Reducing Serialized Size:

If your class has transient or derived fields that need not be serialised, you can exclude them from the serialised form, leading to a smaller serialised size.

It’s worth noting that while**Externalizable** provides more control, it also requires more manual effort to implement, as you need to write custom serialisation and deserialisation logic. In many cases, the default serialisation provided bySerializable is sufficient and more straightforward. However,**Externalizable** offers a powerful alternative for scenarios where customisation is necessary.

Summary :

Serialisation is a fundamental concept in Java that facilitates the storage and transmission of object data. It provides a versatile mechanism for persisting object states, supporting network communication, and enabling interoperability between distributed system components. While the default serialisation mechanism is suitable for many cases, understanding advanced topics like custom serialisation, versioning, and security considerations is essential for developing robust and efficient Java applications.

UseCases for Serialisation in Java

The main reasons for implementing serialisation in Java are as follows:

Persistence :

Serialisation allows objects to be saved to a file and later restored. This is useful for applications that need to store and retrieve the state of objects, such as in database interactions or for caching purposes.

Persistence - Example:

Let’s consider a simple example where we have aPerson class that we want to persist to a file using serialisation.

javaCopy

importjava.io.*;classPersonimplementsSerializable{privateStringname;privateintage;publicPerson(Stringname,intage){this.name=name;this.age=age;}@OverridepublicStringtoString(){return"Person{name='"+name+"', age="+age+'}';}}publicclassSerializationExample{publicstaticvoidmain(String[]args){// Create a Person objectPersonperson=newPerson("John Doe",25);// Serialize the object to a fileserializePerson(person,"person.ser");// Deserialize the object from the filePersondeserializedPerson=deserializePerson("person.ser");// Display the deserialised objectSystem.out.println("Deserialized Person: "+deserializedPerson);}privatestaticvoidserializePerson(Personperson,StringfileName){try(ObjectOutputStreamoutputStream=newObjectOutputStream(newFileOutputStream(fileName))){// Write the object to the fileoutputStream.writeObject(person);System.out.println("Serialization successful. Object saved to "+fileName);}catch(IOExceptione){e.printStackTrace();}}privatestaticPersondeserializePerson(StringfileName){Personperson=null;try(ObjectInputStreaminputStream=newObjectInputStream(newFileInputStream(fileName))){// Read the object from the fileperson=(Person)inputStream.readObject();System.out.println("Deserialization successful. Object loaded from "+fileName);}catch(IOException|ClassNotFoundExceptione){e.printStackTrace();}returnperson;}}

In this example, thePerson class implements theSerializable interface, indicating that its instances can be serialised. TheSerializationExample class demonstrates how to serialise aPerson object to a file (person.ser) and then deserialise it back into a newPerson object. TheserializePerson anddeserializePerson methods handle the serialisation and deserialisation processes, respectively.

Network Communication :

Serialisation enables the easy transmission of objects between different Java Virtual Machines (JVMs) over a network. Converting objects to a byte stream can be sent across a network and reconstructed on the receiving end.

Network Communication - Example:

In this example, we’ll create a simple client-server communication scenario using Java sockets and serialisation. The server will send a serialised object to the client over the network.

Let’s start with the server:

javaCopy

importjava.io.*;importjava.net.ServerSocket;importjava.net.Socket;classPersonimplementsSerializable{privateStringname;privateintage;publicPerson(Stringname,intage){this.name=name;this.age=age;}@OverridepublicStringtoString(){return"Person{name='"+name+"', age="+age+'}';}}publicclassServer{publicstaticvoidmain(String[]args){try{// Create a server socketServerSocketserverSocket=newServerSocket(12345);System.out.println("Server listening on port 12345...");while(true){// Wait for a client to connectSocketclientSocket=serverSocket.accept();System.out.println("Client connected: "+clientSocket.getInetAddress());// Create a Person objectPersonperson=newPerson("Alice",30);// Serialize and send the object to the clientsendObjectToClient(person,clientSocket);// Close the client socketclientSocket.close();}}catch(IOExceptione){e.printStackTrace();}}privatestaticvoidsendObjectToClient(Serializableobj,Socketsocket){try(ObjectOutputStreamoutputStream=newObjectOutputStream(socket.getOutputStream())){// Write the object to the output streamoutputStream.writeObject(obj);System.out.println("Object sent to client.");}catch(IOExceptione){e.printStackTrace();}}}

And here’s the client:

javaCopy

importjava.io.*;importjava.net.Socket;publicclassClient{publicstaticvoidmain(String[]args){try{// Connect to the serverSocketsocket=newSocket("localhost",12345);// Receive the serialised object from the serverPersonreceivedPerson=receiveObjectFromServer(socket);// Display the received objectSystem.out.println("Received Person from server: "+receivedPerson);// Close the socketsocket.close();}catch(IOException|ClassNotFoundExceptione){e.printStackTrace();}}privatestaticPersonreceiveObjectFromServer(Socketsocket)throwsIOException,ClassNotFoundException{try(ObjectInputStreaminputStream=newObjectInputStream(socket.getInputStream())){// Read the object from the input streamreturn(Person)inputStream.readObject();}}}

In this example, theServer class listens for incoming connections on port 12345. When a client connects, it creates aPerson object, serialises it, and sends it to the client using thesendObjectToClient method. TheClient class connects to the server, receives the serialisedPerson object, and then deserialises and prints it.

Object Cloning :

Serialisation provides a way to create a deep copy of an object. By serialising an object and then deserialising it, you effectively create a new copy of the original object. This can be useful when you need to duplicate complex object structures.

Object Cloning - Example:

In this example, I’ll demonstrate object cloning using serialisation in Java. We’ll create aPerson class and use serialisation and deserialisation to perform deep cloning.

javaCopy

importjava.io.*;classAddressimplementsSerializable{privateStringstreet;privateStringcity;publicAddress(Stringstreet,Stringcity){this.street=street;this.city=city;}@OverridepublicStringtoString(){return"Address{street='"+street+"', city='"+city+"'}";}}classPersonimplementsSerializable{privateStringname;privateintage;privateAddressaddress;publicPerson(Stringname,intage,Addressaddress){this.name=name;this.age=age;this.address=address;}@OverridepublicStringtoString(){return"Person{name='"+name+"', age="+age+", address="+address+'}';}}publicclassObjectCloningExample{publicstaticvoidmain(String[]args){// Create a Person objectAddressoriginalAddress=newAddress("123 Main St","Cityville");PersonoriginalPerson=newPerson("John Doe",25,originalAddress);// Clone the object using serialisation and deserialisationPersonclonedPerson=cloneObject(originalPerson);// Modify the original objectoriginalPerson.setName("Jane Doe");originalPerson.setAge(30);originalAddress.setStreet("456 Oak St");// Display the original and cloned objectsSystem.out.println("Original Person: "+originalPerson);System.out.println("Cloned Person: "+clonedPerson);}privatestaticPersoncloneObject(Personoriginal){try{// Serialize the original objectByteArrayOutputStreambos=newByteArrayOutputStream();ObjectOutputStreamout=newObjectOutputStream(bos);out.writeObject(original);out.flush();// Deserialize the object to create a deep copyByteArrayInputStreambis=newByteArrayInputStream(bos.toByteArray());ObjectInputStreamin=newObjectInputStream(bis);return(Person)in.readObject();}catch(IOException|ClassNotFoundExceptione){e.printStackTrace();returnnull;}}}

ThePerson class contains anAddress object in this example. TheObjectCloningExample class demonstrates how to clone aPerson object by serialising it to a byte stream and then deserialising it to create a deep copy. ThecloneObject method handles the serialisation and deserialisation process. After cloning, modifications to the original object do not affect the cloned object, demonstrating a deep copy.

Remote Method Invocation (RMI) :

Java Remote Method Invocation (RMI) is a mechanism that allows objects to invoke methods on objects in another JVM. Serialisation is used to marshal and unmarshal parameters and return values during remote method calls.

JavaBeans :

Serialisation is commonly used in JavaBeans, reusable software components for Java. JavaBeans often need to be serialised to be easily stored or transmitted.

Heads Up

Despite these advantages, it’s important to note that not all classes can or should be serialised. For a class to be serialisable, it must implement theSerializable interface. Additionally, care should be taken when serialising objects to ensure security and proper handling of changes to class definitions over time (versioning).

What are the security issues?

Java Serialization can introduce security issues, particularly when deserialising data from untrusted sources. Here are some of the security concerns associated with Java Serialization:

Remote Code Execution:

One of the most significant security risks with Java Serialization is the potential for remote code execution. When deserialising an object, the Java runtime system can execute arbitrary code within the serialised data. Attackers can exploit this to execute malicious code on the target system. This vulnerability can lead to serious security breaches.

Denial of Service (DoS):

An attacker can create a serialised object with a large size, causing excessive memory consumption and potentially leading to a denial of service attack. Deserialising large objects can consume significant CPU and memory resources, slowing down or crashing the application.

Data Tampering:

Serialised data can be tampered with during transmission or storage. Attackers can modify the serialised byte stream to alter the state of the deserialised object or introduce vulnerabilities.

Insecure Deserialization:

Deserialising untrusted data without proper validation can lead to security issues. For example, if a class that performs sensitive operations is deserialised from untrusted input, an attacker can manipulate the object’s state to perform unauthorised actions.

Information Disclosure:

When objects are serialised, sensitive information may be included in the serialised form. An attacker may gain access to sensitive information if this data needs to be adequately protected or encrypted.

How to Mitigate Serialization Issues

To mitigate these security issues, consider the following best practices:

Avoid Deserializing Untrusted Data:

Avoid deserialising data from untrusted sources altogether. Instead, use safer data interchange formats like JSON or XML for untrusted data.

Implement Input Validation:

When deserialising data, validate and sanitise the input to ensure it adheres to expected data structures and doesn’t contain unexpected or malicious data.

Use Security Managers:

Java’s Security Manager can be used to restrict the permissions and actions of deserialised code. However, it’s important to note that security managers have been removed in newer versions of Java.

Whitelist Classes:

Limit the classes that can be deserialised to a predefined set of trusted classes. This can help prevent the deserialisation of arbitrary and potentially malicious classes.

Versioning and Compatibility:

Be cautious when making changes to serialised classes. UseserialVersionUID to manage versioning and compatibility between different versions of serialised objects.

What is serialVersionUID in Java, and how does it work?

In Java, theserialVersionUID is a unique identifier associated with a serialisable class. It is used during object serialisation to verify that the sender and receiver of a serialised object have loaded classes for that object that are compatible with serialisation. If the receiver has loaded a class for the object with a differentserialVersionUID than the corresponding class on the sender side, deserialisation will result in anInvalidClassException.

Here’s how it works:

Automatic Serialization:

When you mark a class asSerializable in Java, the serialisation mechanism automatically generates a serialVersionUID for that class unless you provide one explicitly. This autogeneratedserialVersionUID is based on various aspects of the class, including its name, implemented interfaces, fields, and methods.

javaCopy

importjava.io.Serializable;publicclassMyClassimplementsSerializable{// class code}

Explicit serialVersionUID:

You can also explicitly declare aserialVersionUID in your class to have more control over versioning. This can be useful to avoid unexpected serialisation compatibility issues when the class structure changes.

javaCopy

importjava.io.Serializable;publicclassMyClassimplementsSerializable{privatestaticfinallongserialVersionUID=123456789L;// class code}

It’s important to note that if you don’t provide an explicitserialVersionUID, the Java runtime will automatically generate one based on the class’s structure, and changes to the class may result in a different autogenerated ID.

Versioning:

When an object is serialised, theserialVersionUID is also stored in the serialised form. During deserialisation, the receiving end compares theserialVersionUID of the loaded class with the one stored in the serialised data. If they match, deserialisation proceeds; otherwise, anInvalidClassException is thrown.

This mechanism helps ensure that the serialised data is compatible with the current class version. If you make changes to the class that are not backwards-compatible, you should update theserialVersionUID to indicate that the new class is incompatible with the previous version.

In summary,serialVersionUID in Java is a version control mechanism for serialised objects, and it plays a crucial role in maintaining compatibility between different versions of a class during object serialisation and deserialisation.

SerialVersionUID Example

Imagine you have aPerson class that represents a person with a name and an age, and you want to serialise instances of this class.

javaCopy

importjava.io.*;publicclassPersonimplementsSerializable{privatestaticfinallongserialVersionUID=1L;privateStringname;privateintage;publicPerson(Stringname,intage){this.name=name;this.age=age;}// getters and setters@OverridepublicStringtoString(){return"Person{name='"+name+"', age="+age+'}';}publicstaticvoidmain(String[]args){// SerializationserializePerson();// DeserializationPersondeserializedPerson=deserializePerson();System.out.println("Deserialized Person: "+deserializedPerson);}privatestaticvoidserializePerson(){try(ObjectOutputStreamoos=newObjectOutputStream(newFileOutputStream("person.ser"))){Personperson=newPerson("John",25);oos.writeObject(person);System.out.println("Person serialized successfully.");}catch(IOExceptione){e.printStackTrace();}}privatestaticPersondeserializePerson(){try(ObjectInputStreamois=newObjectInputStream(newFileInputStream("person.ser"))){return(Person)ois.readObject();}catch(IOException|ClassNotFoundExceptione){e.printStackTrace();returnnull;}}}

In this example:

- ThePerson class implements theSerializable interface.

- It has aserialVersionUID field explicitly set to1L.

- TheserializePerson method creates aPerson object, serialises it, and writes it to a file calledperson.ser.

- ThedeserializePerson method reads and returns the serialisedPerson object from the file.

Now, let’s see how versioning works:

1. Run the program to serialise aPerson object and write it to the file.

2. Change thePerson class by adding a new field, for example,private boolean isStudent;.

3. rerun the program to deserialise thePerson object.

If you don’t update theserialVersionUID when you add the new field, you will likely encounter anInvalidClassException during deserialisation. To avoid this, you should update theserialVersionUID to indicate that the new version of the class is not compatible with the previous one:

javaCopy

**privatestaticfinallongserialVersionUID=2L;**

When you rerun the program, it should deserialise the object successfully, and theisStudent field will have its default value (false). Remember that this is a simple example; in a real-world scenario, you might need to implement more sophisticated versioning strategies based on your application’s requirements.

Security Libraries:

Consider using third-party libraries like Apache Commons Collections or OWASP Java Serialization Security (Java-Serial-Killer) to help mitigate known vulnerabilities and prevent common attacks.

Lessons Learned

In summary, Java Serialization can introduce serious security risks, especially when dealing with untrusted data. It’s essential to take precautions, validate inputs, and consider alternative serialisation methods or libraries to enhance security. Additionally, keeping your Java runtime environment up to date is crucial, as newer versions of Java may include security improvements and fixes for known vulnerabilities.

Conclusion:

Overall, serialisation is both a powerful and dangerous tool. We must balance usage and comfort at each location against our security needs. If you look at open-source projects, you can see a lot of activity to minimise the risk and increase the abstraction of this topic for everyday use.

]]>JavaSecure Coding PracticesSecurityhttps://svenruppert.com/posts/contextual-analysis-in-cybersecurity/Mon, 05 Feb 2024 17:49:29 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/contextual-analysis-in-cybersecurity/Contextual analysis in cybersecurity involves examining events, actions, or data within the broader context of an organization’s IT environment. It is a critical component of a proactive cybersecurity strategy, aiming to understand the significance of activities by considering various factors surrounding them. This multifaceted approach helps cybersecurity professionals identify and respond to potential threats effectively.<![CDATA[

Contextual analysis in cybersecurity involves examining events, actions, or data within the broader context of an organization’s IT environment. It is a critical component of a proactive cybersecurity strategy, aiming to understand the significance of activities by considering various factors surrounding them. This multifaceted approach helps cybersecurity professionals identify and respond to potential threats effectively.

1. Key Concepts
   1. Situational Awareness
      1. Normal Scenario:
      2. Anomalous Scenario:
      3. Situational Awareness Analysis:
      4. Potential Explanations:
      5. Response:
   2. Data Correlation
      1. Scenario:
      2. Data Sources:
      3. Correlation:
      4. Correlated Analysis:
      5. Response:
   3. Threat Intelligence Integration
      1. Scenario:
      2. Threat Intelligence Integration:
      3. Response:
   4. User Behavior Analysis
      1. Scenario:
      2. User Behavior Analysis:
      3. Response:
   5. Temporal Analysis
      1. Scenario:
      2. Analysis:
      3. Contextual Analysis:
      4. Confirmation of Anomaly:
      5. Response:
   6. Environmental Context
      1. Scenario:
      2. Environmental Context Analysis:
      3. Response Based on Environmental Context:
2. Implementation of Contextual Analysis
   1. Security Information and Event Management (SIEM) Systems
   2. Machine Learning and Artificial Intelligence
   3. Automation and Orchestration
   4. Incident Response Planning
3. Challenges and Considerations
   1. False Positives and Negatives
   2. Data Privacy and Compliance
   3. Continuous Monitoring
   4. Skill and Resource Requirements
4. Conclusion

Key Concepts

Situational Awareness

Contextual analysis begins with establishing situational awareness—a comprehensive understanding of an organization’s network, systems, and data. This includes mapping out assets, identifying vulnerabilities, and recognizing standard patterns of behaviour. By understanding the baseline, anomalies and potential threats can be detected more efficiently.

Situational awareness in cybersecurity can be illustrated through an example involving network activity. Imagine a typical day within an organization where employees access company resources and network traffic follows regular patterns.

Normal Scenario:

During working hours, employees log in to their computers, access specific servers, and engage in routine activities such as sending emails, accessing shared files, and using approved applications. During this time, the Network logs consistently show a steady flow of data between internal devices.

Anomalous Scenario:

One day, the network logs indicate an unusually high volume of data being transmitted between an employee’s workstation and a server containing sensitive financial information. This activity is occurring during non-working hours when such data transfers are atypical. Simultaneously, the employee’s account is attempting to access multiple systems that are not part of their regular job responsibilities.

Situational Awareness Analysis:

Situational awareness would involve recognizing the deviation from the baseline of regular network activity. Security analysts would investigate the anomalous behaviour, considering factors such as the timing of the activity, the specific systems involved, and the employee’s role. By correlating data from various sources like logs, endpoint security tools, and user behaviour analytics, the analysts aim to understand the broader context of the situation.

Potential Explanations:

It could be a legitimate activity, such as an employee working on an urgent project during non-working hours. However, this should be validated to ensure it’s not a compromise or misuse of credentials. Alternatively, it might indicate a compromised account, where an external actor or malicious insider is attempting to exfiltrate sensitive data.

Response:

Based on the situational awareness analysis, appropriate responses can be initiated, such as blocking the suspicious activity, notifying the employee to confirm the legitimacy of their actions, and escalating the incident for further investigation if needed. Automated responses, such as temporarily blocking the account, could be triggered to contain potential threats while the situation is assessed.

In this example, situational awareness involves understanding the typical patterns of network activity, identifying anomalies, and responding appropriately to potential security threats based on a broader understanding of the context surrounding the events.

Data Correlation

In a cybersecurity context, data is generated from various sources, such as network logs, endpoint devices, and application activity. The contextual analysis involves correlating this diverse set of data to create a more complete picture of an event or potential threat. Correlation helps distinguish between isolated incidents and coordinated attacks.

Let’s consider an example involving a potential security incident and how data correlation helps in identifying and responding to the threat:

Scenario:

An organization has an SIEM (Security Information and Event Management) system in place, collecting data from different sources such as firewall logs, antivirus alerts, and server logs. One day, the antivirus system generates an alert about a suspicious file being detected on a user’s workstation.

Data Sources:

Firewall Logs : Show an unusual outbound connection from the same user’s workstation to an external IP address.

Antivirus Alert : Indicates the detection of a malware file on the user’s workstation.

Server Logs : Display an increase in failed login attempts on a critical server from the same user’s credentials.

Correlation:

Without data correlation, each event might be treated in isolation, potentially leading to incomplete insights into the security incident. Data correlation involves combining information from these different sources to create a more detailed picture of the situation.

Correlated Analysis:

Firewall Logs + Antivirus Alert : The outbound connection in the firewall logs and the antivirus alert both relate to the same user’s workstation, suggesting that the system might be compromised.

Antivirus Alert + Server Logs : The antivirus alert indicates a malware file on the user’s workstation, and the server logs show an increase in failed login attempts using the same credentials. This correlation suggests that the malware might be attempting to propagate to other systems using stolen credentials.

Firewall Logs + Server Logs : Combining information from firewall logs and server logs reveals that the suspicious outbound connection coincides with the increase in failed login attempts on the critical server. This correlation strengthens the hypothesis that the compromised workstation is attempting unauthorized access to the server.

Response:

With a correlated analysis, security analysts can respond more effectively. They might isolate the compromised workstation from the network, initiate a malware scan, reset the affected user’s credentials, and investigate the attempted unauthorized access to the critical server.

In this example, data correlation helps connect the dots between seemingly isolated security events, enabling a more accurate understanding of the incident. It allows security teams to respond promptly and comprehensively to potential threats by considering the broader context of the data.

Threat Intelligence Integration

Contextual analysis benefits from the integration of threat intelligence. By leveraging information about known threats, vulnerabilities, and attack techniques, organizations can contextualize their security data. This integration enhances the ability to identify and respond to emerging threats that align with known patterns.

Here’s an example illustrating how threat intelligence integration can be applied:

Scenario:

An organization operates in the financial sector and has a robust cybersecurity infrastructure, including an SIEM system and an intrusion detection system (IDS). The organization subscribes to a threat intelligence feed that provides real-time information about emerging cyber threats, including indicators of compromise (IoCs), malware signatures, and tactics, techniques, and procedures (TTPs) associated with specific threat actors.

Threat Intelligence Integration:

New Threat Indicator Detected : The threat intelligence feed identifies a new malicious IP address associated with a known cybercriminal group targeting financial institutions.

Integration with SIEM : The threat intelligence feed is integrated into the organization’s SIEM system. This integration allows the SIEM to update its rules and correlation logic automatically based on the newly identified threat indicator.

SIEM Alerts Triggered : As the SIEM continuously monitors network and system activities, it detects network traffic attempting to communicate with the identified malicious IP address. This triggers alerts within the SIEM system.

Correlation with IDS Alerts : Simultaneously, the IDS generates alerts about suspicious activities that align with the TTPs associated with the cybercriminal group mentioned in the threat intelligence feed.

Analysis and Confirmation : Security analysts, equipped with the integrated threat intelligence, can quickly correlate the SIEM and IDS alerts. They recognize that the detected activities match the known patterns of the cybercriminal group targeting financial institutions.

Response:

With the contextual information from the threat intelligence feed, the security team can tailor their response more effectively. They may block the malicious IP address at the firewall, update antivirus signatures, and implement additional security controls to mitigate the specific threats associated with the identified threat actor.

In this example, threat intelligence integration allows the organization to leverage external insights to fortify its defences and respond swiftly to specific threats. This collaborative approach enhances the organization’s cybersecurity resilience against targeted attacks.

User Behavior Analysis

Understanding typical user behaviour is crucial for contextual analysis. This involves recognizing patterns of activity associated with legitimate users and identifying anomalies that may indicate compromised accounts or malicious insider activities. User behaviour analytics tools play a significant role in this aspect of contextual analysis.

Here’s an example illustrating how user behaviour analysis can be applied:

Scenario:

A medium-sized technology company has implemented user behaviour analysis as part of its cybersecurity strategy. The company’s network includes various departments, each with specific access rights to sensitive data and systems. David, a software developer, typically accesses a particular set of servers and repositories related to his development tasks.

User Behavior Analysis:

Baseline Establishment : The system establishes a baseline of normal behaviour for David. This includes the typical servers he accesses, the times he logs in, and the types of files he interacts with.

Anomaly Detection : One day, the user behaviour analysis system flags an anomaly. David is accessing a server he has never accessed before, and he’s doing so during non-working hours.

Correlation with Other Data : The system correlates this anomaly with other data sources, such as firewall logs and VPN activity. It discovers that, around the same time, there is an unusual outbound connection from David’s workstation to an external IP address.

Contextual Analysis : Security analysts perform contextual analysis. They check the nature of the server David accessed and the destination of the outbound connection. They also cross-reference this with threat intelligence to see if the external IP address is associated with known malicious activity.

Confirmation of Anomaly : Through the analysis, it is confirmed that David’s account has been compromised. An attacker gained access to his credentials, leading to unauthorized access to a server and an attempt to communicate with a potentially malicious external entity.

Response:

The security team takes immediate action. They isolate John’s compromised account, reset his credentials, and investigate the extent of the compromise. Simultaneously, they implement additional security measures, such as two-factor authentication, to prevent future unauthorized access.

In this example, user behaviour analysis assists in identifying a potential security incident involving a compromised user account. By continuously monitoring and analyzing user activities, organizations can enhance their ability to detect and respond to security threats in a timely manner.

Temporal Analysis

Contextual analysis considers the timing of events. Analyzing when certain activities occur can provide insights into their legitimacy. For example, a user accessing sensitive data during non-working hours might raise suspicions. Temporal analysis helps in distinguishing between normal and potentially malicious activities.

Here’s an example illustrating how temporal analysis can be applied:

Scenario:

A financial institution operates a web-based application that facilitates online transactions for its customers. The system logs various events, including user logins, financial transactions, and database access.

Analysis:

Baseline Establishment : The security team establishes a baseline of standard temporal patterns for user logins and financial transactions. For instance, during weekdays, there is a peak in user activity during business hours, and financial transactions are more frequent during specific time windows.

Anomaly Detection : Temporal analysis flags an anomaly: A sudden surge in financial transactions is observed during non-business hours on a weekend. Simultaneously, there’s an unusual pattern of multiple failed login attempts to user accounts.

Correlation with Other Data : The system correlates temporal anomalies with other data sources, such as network logs and intrusion detection system (IDS) alerts. It discovers that the increased financial transactions coincide with a spike in outbound traffic to an unfamiliar IP address.

Contextual Analysis:

Security analysts perform contextual analysis to understand the nature of the financial transactions and the destination of the outbound traffic. They also check if this aligns with known threat indicators or attack patterns.

Confirmation of Anomaly:

Through the analysis, it is confirmed that the temporal anomalies indicate a potential security incident. An attacker has gained unauthorized access to user accounts, attempting fraudulent financial transactions during a time when security defences might be less vigilant.

Response:

The security team takes immediate action to contain the incident. They block unauthorized transactions, reset compromised user accounts, and implement additional controls to prevent further unauthorized access.

In this example, temporal analysis plays a crucial role in detecting unusual patterns of financial transactions and login attempts, allowing the organization to respond promptly to a potential security threat.

Environmental Context

Considering the broader environmental context is essential. This includes factors like the industry in which an organization operates, geopolitical events, and regulatory requirements. Understanding the external context helps in prioritizing threats and aligning cybersecurity efforts with the specific risks an organization faces.

Here’s an example illustrating how environmental context can impact cybersecurity:

Scenario:

A multinational healthcare organization stores sensitive patient information on its servers and operates in a highly regulated industry.

Environmental Context Analysis:

Industry Regulations : The healthcare industry is subject to strict data protection regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States. The organization recognizes that any security incidents leading to data breaches could result in severe legal and financial consequences.

Geopolitical Events : There is a surge in cyber-espionage activities targeting healthcare organizations due to geopolitical tensions. Recent news reports highlight increased hacking attempts and data theft in the industry.

Emerging Cyber Threats : The organization monitors threat intelligence feeds indicating a rise in ransomware attacks targeting healthcare institutions. Recent incidents in the industry have resulted in data encryption and ransom demands.

Internal Changes : The organization is undergoing a digital transformation, implementing new technologies to enhance patient care. This introduces additional attack surfaces and potential vulnerabilities that need to be addressed.

Response Based on Environmental Context :

Recognizing the environmental context, the organization takes several strategic cybersecurity measures:

Enhanced Regulatory Compliance : The cybersecurity strategy places a strong emphasis on compliance with healthcare regulations. The organization invests in solutions that specifically address HIPAA requirements, such as encryption and access controls.

Geopolitical Threat Mitigation : The security team increases monitoring and implements additional controls to defend against cyber-espionage activities targeting healthcare organizations. They collaborate with industry peers to share threat intelligence and best practices.

Ransomware Preparedness : Given the rising threat of ransomware, the organization conducts regular training for employees on recognizing phishing attempts and establishes robust backup and recovery processes to minimize the impact of a potential ransomware attack.

Adaptive Security Measures : The cybersecurity strategy acknowledges the evolving nature of the organization’s digital landscape. Security controls are regularly updated to accommodate new technologies introduced during the digital transformation, ensuring a proactive approach to emerging threats.

In this example, environmental context plays a pivotal role in shaping the organization’s cybersecurity strategy, allowing it to effectively navigate industry-specific challenges and emerging threats.

Implementation of Contextual Analysis

Security Information and Event Management (SIEM) Systems

SIEM systems play a central role in contextual analysis by aggregating and analyzing security data from various sources. These platforms use correlation rules and behavioural analytics to identify potential threats. The integration of threat intelligence feeds further enhances their ability to provide context to security events.

It is a comprehensive solution designed to provide a holistic view of an organization’s information security. It combines Security Information Management (SIM) and Security Event Management (SEM) into one integrated platform.

The primary goal of an SIEM system is to provide real-time analysis of security alerts generated throughout an organization’s IT infrastructure.

Here are the key components and functionalities of an SIEM system:

Data Collection : SIEM systems collect and aggregate log data generated throughout the organization’s technology infrastructure, from host systems and applications to network and security devices.

Normalization and Correlation : The collected data is normalized to a standard format, allowing for more straightforward analysis. SIEM tools also correlate related events to help identify patterns and potential security threats.

Alerting and Notification : SIEM systems can generate alerts in real-time or near real-time, notifying security administrators of potential security incidents or policy violations. Alerts are often prioritized based on severity.

Dashboards and Reporting : SIEM solutions provide dashboards and reports that offer insights into the security status of an organization. These visualizations help security professionals understand the overall security posture, trends, and potential areas of concern.

Incident Response : SIEM systems assist in incident response by providing detailed information about security incidents, helping security teams investigate and mitigate threats more effectively.

Compliance Monitoring : Many organizations use SIEM tools to meet regulatory compliance requirements by monitoring and reporting on activities that are subject to specific regulations.

User and Entity Behavior Analytics (UEBA) : Some SIEM solutions incorporate UEBA, which uses machine learning and statistical analysis to detect abnormal behaviour patterns among users and entities, helping to identify potential insider threats.

Integration with Other Security Tools : SIEM systems often integrate with other security solutions, such as intrusion detection/prevention systems, antivirus software, and vulnerability management tools, to provide a more comprehensive security posture.

Log Retention and Storage : SIEM tools typically store log and event data for an extended period, allowing for historical analysis and compliance reporting.

By centralizing and analyzing security event data from across an organization’s technology infrastructure, SIEM systems play a crucial role in enhancing an organization’s ability to detect and respond to security incidents effectively. They are an essential component of a robust cybersecurity strategy.

Machine Learning and Artificial Intelligence

Machine learning algorithms contribute significantly to contextual analysis. These technologies can analyze vast amounts of data, recognize patterns, and identify anomalies that may elude traditional rule-based systems. AI-driven tools excel in adapting to evolving threats by continuously learning from new data.

Automation and Orchestration

Automated responses to security incidents are a natural extension of contextual analysis. When an anomalous event is detected, automatic responses can be triggered to contain or mitigate the threat. Orchestration ensures that these automated actions are coordinated across different security tools and systems.

Incident Response Planning

Contextual analysis is an integral part of incident response planning. Organizations should have predefined workflows that consider the context of an incident. This includes understanding the potential impact, the criticality of affected systems, and the broader implications for the business.

Challenges and Considerations

False Positives and Negatives

One of the challenges in contextual analysis is the balance between minimizing false positives (incorrectly identifying benign events as threats) and avoiding false negatives (failing to detect actual threats). Achieving this balance requires fine-tuning correlation rules and leveraging advanced analytics.

Data Privacy and Compliance

As contextual analysis involves analyzing diverse data sources, ensuring compliance with data privacy regulations is crucial. Organizations must strike a balance between practical analysis and respecting the privacy rights of individuals.

Continuous Monitoring

Contextual analysis is an ongoing process that requires continuous monitoring and adaptation. Cybersecurity threats evolve, and so should contextual analysis strategies. Regularly updating correlation rules, threat intelligence feeds and adjusting algorithms is essential to stay ahead of emerging threats.

Skill and Resource Requirements

Implementing practical contextual analysis requires skilled cybersecurity professionals and adequate resources. Organizations need personnel who can interpret the context of security events and make informed decisions. Additionally, investing in advanced technologies and training programs is essential.

Conclusion

Contextual analysis is a dynamic and integral part of modern cybersecurity strategies. By understanding the broader context of security events, organizations can enhance their ability to detect, respond to, and mitigate cyber threats effectively. As cyber threats continue to evolve, contextual analysis will play a pivotal role in securing digital assets and maintaining the resilience of organizations in the face of cyber challenges.

]]>Securityhttps://svenruppert.com/posts/what-is-a-common-weakness-enumeration-cwe/Wed, 10 Jan 2024 17:24:15 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/what-is-a-common-weakness-enumeration-cwe/CWE stands for Common Weakness Enumeration. It is a community-developed list of software and hardware weakness types that can serve as a common language for describing, sharing, and identifying security vulnerabilities in software systems. CWE aims to provide a standardized way of identifying and categorizing vulnerabilities, making it easier for software developers, testers, and security professionals to discuss and address security issues.<![CDATA[

CWE stands for Common Weakness Enumeration. It is a community-developed list of software and hardware weakness types that can serve as a common language for describing, sharing, and identifying security vulnerabilities in software systems. CWE aims to provide a standardized way of identifying and categorizing vulnerabilities, making it easier for software developers, testers, and security professionals to discuss and address security issues.

1. Why Do We Need A Common Weakness Enumeration?
   1. Standardized Language:
   2. Improved Awareness:
   3. Efficient Communication:
   4. Systematic Analysis:
   5. Security Education and Training:
   6. Tool Integration:
   7. Vulnerability Management:
   8. Community Collaboration:
   9. Regulatory Compliance:
2. Details about Common Weakness Enumeration
   1. Development and Maintenance:
   2. Standardized Identifiers:
   3. Categories and Views:
   4. Relationships and Mappings:
   5. Community Involvement:
   6. Software Assurance:
   7. Integration with other Standards:
   8. Education and Training:
3. The main components of the CWE structure:
4. An Example for the Java-World
   1. CWE-129: Improper Validation of Array Index
      1. Description:
      2. Example in Java:
5. A List Of Example CWE´s

Why Do We Need A Common Weakness Enumeration?

Common Weakness Enumeration (CWE) serves several essential purposes in cybersecurity and software development. Here are some key reasons why CWE is needed:

Standardized Language:

CWE provides a standardized and common language for describing and categorizing software and hardware weaknesses. This common terminology helps improve communication among developers, testers, and security professionals, fostering a shared understanding of vulnerabilities.

Improved Awareness:

CWE raises awareness about common weaknesses and vulnerabilities in software systems. This awareness is essential for developers and security practitioners to proactively identify and address potential issues during the software development life cycle.

Efficient Communication:

Using unique identifiers (CWE IDs) for each weakness allows for efficient communication and reference. When discussing vulnerabilities, using CWE IDs ensures clarity about which specific weakness is being referred to.

Systematic Analysis:

CWE organizes weaknesses into categories and views, enabling a systematic analysis of vulnerabilities. This organization helps identify patterns and trends in security issues, allowing for more effective risk assessment and mitigation strategies.

Security Education and Training:

CWE is used as a foundation for security education and training programs. Developers can enhance their skills and knowledge by understanding common weaknesses, bettering them to write secure code and design robust systems.

Tool Integration:

Many security tools and systems use CWE to categorize and report vulnerabilities. This integration streamlines identifying and remedying weaknesses by providing a standardized way for different tools to communicate and share information about security issues.

Vulnerability Management:

CWE contributes to effective vulnerability management by providing a comprehensive list of weaknesses. This allows organizations to prioritize and address vulnerabilities based on their severity and potential impact on the system.

Community Collaboration:

The collaborative development and maintenance of CWE involve input from a wide range of stakeholders, including researchers, industry professionals, and organizations. This collaborative approach ensures that CWE remains comprehensive, relevant, and reflective of the evolving landscape of software vulnerabilities.

Regulatory Compliance:

Some regulatory frameworks and standards reference or require CWE to identify and manage software weaknesses. Compliance with these standards may be necessary for organizations operating in specific industries or regions.

In summary, Common Weakness Enumeration is crucial in enhancing cybersecurity practices by providing a standardized and widely accepted framework for identifying, categorizing, and addressing software weaknesses and vulnerabilities. It contributes to a more secure software development and deployment ecosystem by promoting awareness, communication, and collaboration within the cybersecurity community.

Details about Common Weakness Enumeration

Development and Maintenance:

CWE is developed and maintained by the MITRE Corporation, a not-for-profit organization that operates Federally Funded Research and Development Centers (FFRDCs) in the United States.

Standardized Identifiers:

Each weakness in the CWE list is assigned a unique CWE ID identifier. This helps in unambiguously referring to specific weaknesses when discussing vulnerabilities.

Categories and Views:

CWE organizes weaknesses into categories and views. Categories group similar weaknesses together, making understanding and addressing issues systematically easier. Views provide different perspectives on the data, allowing users to focus on specific aspects of weaknesses.

Relationships and Mappings:

CWE captures relationships between weaknesses, showing how they might be related or lead to other vulnerabilities. It also provides mappings to other relevant standards and frameworks, facilitating integration with different tools and processes.

Community Involvement:

The development of CWE involves collaboration with the broader cybersecurity community, including researchers, developers, and security professionals. Input is collected to ensure that the list remains comprehensive and up-to-date.

Software Assurance:

CWE is part of a broader initiative focused on improving software assurance. Providing a common language for describing weaknesses contributes to building more secure and robust software systems.

Integration with other Standards:

CWE is often used alongside standards like the Common Vulnerability Scoring System (CVSS) and the Common Vulnerability and Exposure (CVE) system. Together, These standards help assess, score, and address vulnerabilities in a coordinated manner.

Education and Training:

CWE is used for educational purposes, helping individuals and organizations better understand common software weaknesses and vulnerabilities. Training materials and resources are developed to support this educational aspect.

In summary, Common Weakness Enumeration plays a crucial role in standardizing the identification and classification of software weaknesses, providing a foundation for improving cybersecurity practices across the software development and security communities.

The main components of the CWE structure:

Entry Point :

You have the entry points at the top level of the hierarchy. These represent high-level categories or groups of weaknesses. Examples of entry points include “CWE-119: Improper Restriction of Operations within the Bounds of a Memory Buffer” and “CWE-200: Exposure of Sensitive Information to an Unauthorized Actor.”

Categories :

Entry points are further divided into categories. Categories group related weaknesses together based on similar characteristics or types of vulnerabilities. For example, under “CWE-119,” you might have categories like “CWE-120: Buffer Copy without Checking Size of Input (‘Classic Buffer Overflow’)” or “CWE-121: Stack-based Buffer Overflow.”

Weaknesses :

Categories are broken down into specific weaknesses. Each weakness has a unique (CWE ID) identifier and detailed description. For example, under “CWE-120,” you might have particular weaknesses like “CWE-120: Classic Buffer Overflow in strcat()” or “CWE-121: Stack-based Buffer Overflow in strcpy()”.

Relationships :

CWE also captures relationships between weaknesses. This includes hierarchical relationships, where a more general weakness encompasses more specific weaknesses. Additionally, it identifies other types of relationships, such as dependencies, mappings to other standards, and variations.

Views :

CWE provides different views to offer alternative perspectives on the data. Views may focus on specific aspects of weaknesses, and users can switch between views to examine the information from different angles.

Mitigations :

Each weakness entry includes information on potential mitigations or ways to address and prevent the weakness. This helps users understand how to secure their software against specific vulnerabilities.

An Example for the Java-World

One common weakness is the improper validation of array indices, which can lead to various security issues, including buffer overflows. CWE-129 specifically addresses this issue.

CWE-129: Improper Validation of Array Index

Description:

Improper validation of array indices in Java can lead to out-of-bounds read or write access, potentially causing unexpected behaviour, crashes, or security vulnerabilities.

Example in Java:

javaCopy

publicclassArrayIndexExample{publicstaticvoidmain(String[]args){int[]numbers={1,2,3,4,5};// Incorrect: Accessing an array element without proper bounds checkingintindex=10;intvalue=numbers[index];// This can lead to ArrayIndexOutOfBoundsException// Correct: Adding proper bounds checking to prevent array index issuesif(index>=0&&index<numbers.length){value=numbers[index];System.out.println("Value at index "+index+": "+value);}else{System.out.println("Invalid index: "+index);}}}

In this example, the improper validation of the array index occurs when trying to access an element at the index**10** without checking whether it is within the array’s bounds. This can result in an**ArrayIndexOutOfBoundsException** at runtime. The corrected version includes proper bounds checking to ensure the index is within the valid range before accessing the array.

Addressing CWEs like this one is crucial for writing secure Java code and avoiding common vulnerabilities related to array index manipulation. Keep in mind that this is just one example, and there are many other CWEs that developers should be aware of and mitigate in their Java applications.

A List Of Example CWE´s

Certainly! Below is a list of Common Weakness Enumeration (CWE) entries that are commonly associated with security issues in Java applications. Each entry includes a brief description:

CWE-20: Improper Input Validation

Failure to properly validate user-supplied input can lead to various security vulnerabilities, including injection attacks (e.g., SQL injection, command injection).

CWE-79: Improper Neutralization of Input During Web Page Generation (‘Cross-site Scripting’)

Failure to properly sanitize user input before rendering it on a web page can lead to Cross-Site Scripting (XSS) vulnerabilities.

CWE-89: Improper Neutralization of Special Elements used in an SQL Command (‘SQL Injection’)

Failure to properly validate and sanitize input used in SQL queries can result in SQL injection vulnerabilities.

CWE-77: Improper Neutralization of Special Elements used in a Command (‘Command Injection’)

Like SQL injection, improper user input validation in command-based operations can lead to command injection vulnerabilities.

CWE-352: Cross-Site Request Forgery (CSRF)

Inadequate validation of requests can expose applications to Cross-Site Request Forgery attacks, where unauthorized actions are performed on behalf of a user.

CWE-306: Missing Authentication for Critical Function

Failing to enforce proper authentication for critical functions can lead to unauthorized access and potential security breaches.

CWE-285: Improper Authorization

Inadequate enforcement of access controls and permissions can result in unauthorized access to sensitive data or functionality.

CWE-311: Missing Encryption of Sensitive Data

Failure to encrypt sensitive data during storage or transmission can lead to data exposure and privacy issues.

CWE-732: Incorrect Permission Assignment for Critical Resource

Assigning incorrect permissions to critical resources can result in unauthorized access and potential security vulnerabilities.

CWE-400: Uncontrolled Resource Consumption (‘Resource Exhaustion’)

Lack of proper resource management can lead to resource exhaustion attacks, where an attacker consumes system resources to disrupt service availability.

CWE-601: URL Redirection to Untrusted Site (‘Open Redirect’)

Improper validation of user-supplied input in URL redirection can lead to open redirect vulnerabilities, potentially facilitating phishing attacks.

CWE-502: Deserialization of Untrusted Data

Insecure deserialization of untrusted data can lead to remote code execution and other security vulnerabilities.

It’s important to note that the information provided here is a brief overview, and developers should refer to the official CWE website or other authoritative sources for more in-depth information and guidance on mitigating these vulnerabilities in Java applications.

]]>Secure Coding PracticesSecurityhttps://svenruppert.com/posts/secure-coding-practices-input-validation/Wed, 13 Dec 2023 07:52:24 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/secure-coding-practices-input-validation/What is - Input Validation? Input validation is a process used to ensure that the data provided to a system or application meets specific criteria or constraints before it is accepted and processed. The primary goal of input validation is to improve the reliability and security of a system by preventing invalid or malicious data from causing errors or compromising the system’s integrity.<![CDATA[

What is - Input Validation?

Input validation is a process used to ensure that the data provided to a system or application meets specific criteria or constraints before it is accepted and processed. The primary goal of input validation is to improve the reliability and security of a system by preventing invalid or malicious data from causing errors or compromising the system’s integrity.

1. What is - Input Validation?
2. What are the critical aspects of input validation:
   1. Data Integrity:
   2. Security:
   3. User Experience:
   4. Compliance:
   5. Preventing Bugs:
3. What does input validation have to do with security?
   1. Preventing Injection Attacks:
   2. Avoiding Buffer Overflows:
   3. Mitigating Cross-Site Request Forgery (CSRF) Attacks:
   4. Protecting Against Data Tampering:
   5. Enhancing Authentication and Authorisation:
   6. Securing File Operations:
   7. Preventing Denial-of-Service (DoS) Attacks:
   8. Ensuring Data Integrity:
4. Where can input validation take place in a program?
   1. User Interface (UI) Level:
   2. Server-Side Level:
   3. Database Level:
   4. API Level:
   5. Middleware Level:
   6. Configuration Files:
   7. Logging Level:
   8. Web Application Firewall (WAF):
5. What are the advantages and disadvantages of server-side and client-side input validation?
   1. Server-Side Input Validation
      1. Advantages:
         1. Security:
         2. Data Integrity:
         3. Business Logic:
         4. User Experience:
         5. Cross-Browser Compatibility:
         6. Code Reusability:
      2. Disadvantages:
         1. Response Time:
         2. User Experience:
         3. Bandwidth Usage:
         4. Server Load:
   2. Client-Side Input Validation
      1. Advantages:
         1. Immediate Feedback:
         2. Bandwidth Efficiency:
         3. Enhanced User Experience:
         4. Reduced Server Load:
         5. Reduced Round-Trip Time:
      2. Disadvantages:
         1. Security Risks:
         2. Browser Dependency:
         3. Code Duplication:
         4. Accessibility:
         5. Browser Performance:
   3. Lessons Learned - client- versus server-side validation:
6. What are standard techniques for input validation?
   1. Data Type Checks:
   2. Length Checks:
   3. Format Checks:
   4. Range Checks:
   5. Pattern Matching:
   6. Allowlisting/Blocklisting:
7. Conclusion:

What are the critical aspects of input validation:

Data Integrity:

Input validation helps maintain data integrity by ensuring it conforms to the expected format and type. For example, if a system expects a numerical input, input validation can check whether the entered data is a number.

Security:

Proper input validation is crucial for preventing security vulnerabilities such as injection attacks (e.g., SQL injection, cross-site scripting) where malicious data is inserted to exploit vulnerabilities in a system.

User Experience:

Input validation can also contribute to a better user experience by providing immediate feedback to users about the correctness of their input. Instead of allowing users to submit invalid data and encounter errors, input validation helps users correct mistakes upfront.

Compliance:

In some industries, regulatory requirements mandate proper input validation to protect sensitive information and ensure the appropriate functioning of systems.

Preventing Bugs:

Input validation can help prevent bugs and unexpected behaviour in software by ensuring that the data flowing through the system adheres to the expected standards.

What does input validation have to do with security?

Input validation is crucial for security in software development because it helps prevent a wide range of security vulnerabilities and attacks. Here are some key reasons why input validation is essential for security:

Preventing Injection Attacks:

Input validation helps protect against injection attacks, where malicious code is injected into user input. Common examples include SQL injection and cross-site scripting (XSS) attacks. By validating and sanitising input, developers can ensure that user input doesn’t contain malicious code the application could execute.

Avoiding Buffer Overflows:

Input validation helps prevent buffer overflows, a vulnerability where an attacker can exploit the program’s memory by providing input that exceeds the allocated buffer size. By validating input size and content, developers can mitigate the risk of buffer overflow attacks.

Mitigating Cross-Site Request Forgery (CSRF) Attacks:

CSRF attacks involve tricking a user’s browser into making unintended requests to a web application on which the user is authenticated. Proper input validation ensures that requests originate from the expected sources, reducing the risk of CSRF attacks.

Protecting Against Data Tampering:

Input validation helps protect against data tampering by ensuring that the data received by the application adheres to expected formats and constraints. This prevents attackers from manipulating data to exploit vulnerabilities or gain unauthorised access.

Enhancing Authentication and Authorisation:

Validating input is essential for enforcing proper authentication and authorisation checks. Incorrect or manipulated input could bypass authentication mechanisms or grant unauthorised access to sensitive areas of an application.

Securing File Operations:

Input validation is crucial when dealing with file uploads or file-related operations. Without proper validation, an attacker might upload malicious files or manipulate file names to execute arbitrary code on the server.

Preventing Denial-of-Service (DoS) Attacks:

Input validation can mitigate the impact of DoS attacks by rejecting or limiting input that could lead to resource exhaustion. For example, limiting the length of input or the rate of requests can help protect against certain types of DoS attacks.

Ensuring Data Integrity:

Input validation contributes to data integrity by ensuring that the data processed by an application meets expected standards. This is important for preventing unintentional errors and maintaining the consistency and reliability of the system.

In summary, input validation is a fundamental security practice that helps build robust and resilient software systems. By validating and sanitising input data, developers can significantly reduce the risk of common security vulnerabilities and enhance the overall security posture of their applications.

Where can input validation take place in a program?

Input validation is a crucial aspect of writing secure and robust software. It helps ensure that the data your program receives is of the expected format and within acceptable ranges. Input validation can take place at various levels in a program, including:

User Interface (UI) Level:

Client-Side Validation: It can be performed using JavaScript or other client-side scripting languages in web applications to provide immediate feedback to users. However, client-side validation should not be solely relied upon for security, as malicious users can bypass it.

Server-Side Level:

Server-Side Validation: This is the most critical layer for input validation. All input received from the client should be validated on the server to ensure that it conforms to the expected format, length, and other criteria. Server-side validation is more secure than client-side validation because it cannot be easily bypassed or tampered with by users.

Database Level:

Database Input Validation: Before storing data in a database, validating the input is essential to prevent SQL injection attacks and ensure data integrity. This can involve using parameterised queries or prepared statements to prevent malicious input from affecting the database.

API Level:

API Input Validation: If your application interacts with external APIs, it’s crucial to validate input data before sending requests. This helps prevent injection attacks and ensures the API receives the expected data.

Middleware Level:

Middleware Validation: Middleware components may process or filter incoming data in some applications before it reach the core application logic. Input validation can be performed at this level to ensure only valid data is passed to the application.

Configuration Files:

Configuration Input Validation: If your program reads configuration files, validate the input data to ensure it adheres to the expected format and doesn’t introduce vulnerabilities.

Logging Level:

Logging Input Validation: When logging user inputs or other sensitive information, ensure that the logged data is properly validated to prevent accidental exposure of sensitive information or injection attacks through log manipulation.

Web Application Firewall (WAF):

WAF Validation: WAFs can filter and validate incoming HTTP traffic. They can help protect against common web application vulnerabilities, including input-related attacks.

By implementing input validation at multiple levels, you create a defence-in-depth strategy that enhances the security of your application. Always validate input data based on your application’s requirements and constraints to ensure proper functionality and security.

What are the advantages and disadvantages of server-side and client-side input validation?

At this point, I’ll briefly overview the advantages and disadvantages of server-side and client-side input validation, covering various aspects of security, performance, user experience, and maintenance.

Server-Side Input Validation

Advantages:

Security:

Protection Against Malicious Input : Server-side validation is crucial for security as it prevents malicious input from reaching the database or application logic. This is especially important for avoiding SQL injection, cross-site scripting (XSS), and other security vulnerabilities.

Centralised Control : Centralised validation on the server allows for consistent and thorough input data validation, reducing the risk of overlooking potential security threats.

Data Integrity:

Consistent Data Quality : Server-side validation ensures that the data entering the system meets the specified criteria, maintaining data integrity and consistency throughout the application.

Prevention of Data Corruption : The risk of corrupted or invalid data entering the database is minimised by validating input on the server.

Business Logic:

Enforcement of Business Rules : Server-side validation is essential for enforcing business rules and logic. This ensures that data conforms not only to technical specifications but also to the application’s specific requirements.

User Experience:

Consistent Validation Messages : Server-side validation provides the opportunity to generate standardised error messages, creating a more consistent and user-friendly experience for end-users.

Cross-Browser Compatibility:

Browser Independence : Server-side validation is not dependent on the user’s browser, making it more reliable and consistent across different browser environments.

Code Reusability:

Centralised Validation Logic : Server-side validation allows for the centralisation of validation logic, promoting code reusability across multiple application parts.

Disadvantages:

Response Time:

Increased Round-Trip Time : Server-side validation requires a round-trip to the server, leading to potential delays in response time, especially in applications with high latency.

User Experience:

Delayed Feedback : Users may experience delays in receiving feedback on their input, leading to a less responsive and potentially frustrating user experience.

Bandwidth Usage:

Increased Bandwidth Usage : Each validation request sent to the server consumes bandwidth, which can be a concern in bandwidth-sensitive environments or for users with limited data plans.

Server Load:

Increased Server Load : Server-side validation contributes to server load, and in high-traffic applications, this can lead to resource contention and decreased overall performance.

Client-Side Input Validation

Advantages:

Immediate Feedback:

Real-time Validation : Client-side validation provides immediate feedback to users, improving the application’s overall responsiveness and perceived speed.

Bandwidth Efficiency:

Reduced Server Requests : Client-side validation minimises the number of requests to the server, conserving bandwidth and improving overall network efficiency.

Enhanced User Experience:

Responsive Interactivity : By validating input on the client side, users receive instant feedback, creating a more interactive and responsive user experience.

Reduced Server Load:

Offloading Server Processing : Client-side validation offloads some processing tasks from the server, reducing the overall load and improving server performance.

Reduced Round-Trip Time:

Faster Form Submissions : Since validation occurs on the client side, users experience speedier form submissions without waiting for a round-trip to the server.

Disadvantages:

Security Risks:

Vulnerability to Manipulation : Client-side validation is susceptible to manipulation by malicious users who can bypass or disable the client-side validation to submit malicious input directly to the server.

Browser Dependency:

Inconsistent Browser Support : Client-side validation may need consistent support across different browsers, leading to behaviour and user experience variations.

Code Duplication:

Duplication of Validation Logic : Client-side validation requires duplicating validation logic on the server to ensure data integrity and security, leading to potential maintenance challenges.

Accessibility:

Accessibility Concerns : Relying solely on client-side validation may pose accessibility challenges for disabled users using alternative input methods or assistive technologies.

Browser Performance:

Resource Intensive : Intensive client-side validation logic can consume significant browser resources, impacting the performance of the user’s device.

Lessons Learned - client- versus server-side validation:

In conclusion, the choice between server-side and client-side input validation depends on various factors such as security requirements, user experience goals, and application architecture. A balanced approach often involves a combination of both server-side and client-side validation to harness the advantages of each while mitigating their respective disadvantages. Careful consideration of the specific needs and constraints of the application is crucial to implementing a practical and secure input validation strategy.

What are standard techniques for input validation?

Data Type Checks:

Ensuring that the entered data matches the expected data type (e.g., numbers, dates, strings).

Here’s a simple Java example using theinstanceof operator, which checks whether an object is an instance of a particular class or interface:

javaCopy

publicclassDataTypeCheckExample{publicstaticvoidmain(String[]args){// Example with an IntegerObjectvalue=42;if(valueinstanceofInteger){intintValue=(Integer)value;System.out.println("The value is an Integer: "+intValue);}else{System.out.println("The value is not an Integer.");}// Example with a StringObjectanotherValue="Hello, Java!";if(anotherValueinstanceofString){StringstringValue=(String)anotherValue;System.out.println("The value is a String: "+stringValue);}else{System.out.println("The value is not a String.");}}}

In this example, we have anObject variable (value andanotherValue) that can hold any object. We use theinstanceof operator to check whether the object is an instance of a specific type (Integer orString). We can safely cast it to that type and perform operations accordingly if it is. If not, we handle it appropriately.

Remember, it’s generally a good practice to use proper data types whenever possible rather than relying on type checks, but there are situations where type checks are necessary.

Length Checks:

Verifying that the length of the input is within acceptable limits.

In Java, you can perform length checks on various data structures, such as strings or arrays, to ensure they meet specific criteria. Here’s an example of length checks for a string in Java:

javaCopy

publicclassLengthCheckExample{publicstaticvoidmain(String[]args){//example stringStringmyString="Hello, World!";// Check the length of the stringintlength=myString.length();// Perform length checksif(length>0){System.out.println("The string is not empty.");}else{System.out.println("The string is empty.");}if(length>=10){System.out.println("The string is at least 10 characters long.");}else{System.out.println("The string is less than 10 characters long.");}}}

In this example, we use thelength() method of theString class to get the length of the string. We then perform two length checks: one to determine if the string is empty and another to check if it’s at least ten characters long.

Remember that the specific length of checks you perform will depend on the requirements of your program or application. You can adapt similar checks for arrays or other data structures as needed.

Format Checks:

Ensuring the input follows a specified format (e.g., email addresses, phone numbers).

Below is an example Java program that performs format checks for email addresses and phone numbers using regular expressions:

javaCopy

importjava.util.regex.Matcher;importjava.util.regex.Pattern;publicclassFormatChecker{publicstaticvoidmain(String[]args){// Example usageStringemail="user@example.com";StringphoneNumber="+1-555-555-5555";if(isValidEmail(email)){System.out.println("Email is valid: "+email);}else{System.out.println("Email is not valid: "+email);}if(isValidPhoneNumber(phoneNumber)){System.out.println("Phone number is valid: "+phoneNumber);}else{System.out.println("Phone number is not valid: "+phoneNumber);}}// Check if the email address is validpublicstaticbooleanisValidEmail(Stringemail){StringemailRegex="^[a-zA-Z0-9_+&*-]+(?:\\.[a-zA-Z0-9_+&*-]+)*@(?:[a-zA-Z0-9-]+\\.)+[a-zA-Z]{2,7}$";Patternpattern=Pattern.compile(emailRegex);Matchermatcher=pattern.matcher(email);returnmatcher.matches();}// Check if the phone number is validpublicstaticbooleanisValidPhoneNumber(StringphoneNumber){// Accepts numbers in the format: +1-555-555-5555StringphoneRegex="^\\+(?:[0-9] ?){6,14}[0-9]$";Patternpattern=Pattern.compile(phoneRegex);Matchermatcher=pattern.matcher(phoneNumber);returnmatcher.matches();}}

In this example,isValidEmail andisValidPhoneNumber are two methods that use regular expressions to check if the provided email address and phone number are in the expected format. The regular expressions in this example may need adjustment based on your specific format requirements. Regular expressions are powerful tools for pattern matching but can be complex. It’s essential to thoroughly test them with various inputs to ensure they meet your needs.

If you prefernot to use regular expressions, you can perform format checks for email addresses and phone numbers using other string manipulation techniques. Here’s an example:

javaCopy

publicclassFormatChecker{publicstaticvoidmain(String[]args){// Example usageStringemail="user@example.com";StringphoneNumber="+1-555-555-5555";if(isValidEmail(email)){System.out.println("Email is valid: "+email);}else{System.out.println("Email is not valid: "+email);}if(isValidPhoneNumber(phoneNumber)){System.out.println("Phone number is valid: "+phoneNumber);}else{System.out.println("Phone number is not valid: "+phoneNumber);}}// Check if the email address is validpublicstaticbooleanisValidEmail(Stringemail){intatIndex=email.indexOf('@');intdotIndex=email.lastIndexOf('.');// Ensure there is exactly one @ symbol, and it is not the first or last characterif(atIndex>0&&atIndex<email.length()-1){// Ensure there is at least one dot after the @ symbolif(dotIndex>atIndex&&dotIndex<email.length()-1){returntrue;}}returnfalse;}// Check if the phone number is validpublicstaticbooleanisValidPhoneNumber(StringphoneNumber){// Check the length and specific charactersif(phoneNumber.length()>=10&&phoneNumber.charAt(0)=='+'&&phoneNumber.charAt(1)>='1'&&phoneNumber.charAt(1)<='9'){// Check that the rest of the characters are digits or hyphensfor(inti=2;i<phoneNumber.length();i++){charc=phoneNumber.charAt(i);if(!(Character.isDigit(c)||c=='-')){returnfalse;}}returntrue;}returnfalse;}}

This example uses basic string manipulation and checks for specific conditions rather than regular expressions. Remember that regular expressions are often a more concise and powerful way to perform such checks, but this alternative approach may be suitable for more straightforward cases.

Range Checks:

Verifying that numerical input falls within a predefined range.

You can perform range checks using conditional statements. Here’s a simple example that demonstrates range checks for a given value within a specified range:

javaCopy

publicclassRangeCheckExample{publicstaticvoidmain(String[]args){// Define the rangeintlowerBound=10;intupperBound=20;// Test valuesintvalue1=15;intvalue2=5;intvalue3=25;// Perform range checksSystem.out.println("Value "+value1+" is in range: "+isInRange(value1,lowerBound,upperBound));System.out.println("Value "+value2+" is in range: "+isInRange(value2,lowerBound,upperBound));System.out.println("Value "+value3+" is in range: "+isInRange(value3,lowerBound,upperBound));}// Function to check if a value is within a specified rangeprivatestaticbooleanisInRange(intvalue,intlowerBound,intupperBound){returnvalue>=lowerBound&&value<=upperBound;}}

In this example, theisInRange function takes a value and two bounds (lower and upper). It returnstrue if the value is within the specified range (inclusive) andfalse otherwise. Themain method demonstrates the usage of this function for three different values and prints whether each value is in the specified range.

Pattern Matching:

Checking input against a predefined pattern or regular expression.

Pattern matching was introduced in Java 16 as a preview feature and became standard in Java 17. Here’s a simple example using pattern matching with the oldinstanceof operator:

javaCopy

publicclassPatternMatchingExample{publicstaticvoidmain(String[]args){Objectvalue="Hello, Pattern Matching!";if(valueinstanceofStringstr){System.out.println("Length of the string: "+str.length());}else{System.out.println("Not a string");}}}

In this example, we have anObject calledvalue, and we want to check if it is an instance ofString. If it is, we introduce a new variablestr of typeString in the sameif statement and can use it within that block. This eliminates the need for an additional cast and makes the code more concise.

In Java 16, pattern matching was introduced as a preview feature, and whileinstanceof was commonly used, there were no specific pattern-matching constructs. However, one common pattern-matching use case was inswitch expressions. Here’s an example:

javaCopy

publicclassPatternMatchingExample{publicstaticvoidmain(String[]args){Objectvalue="Hello, Pattern Matching!";intresult=switch(value){caseStringstr->str.length();caseIntegernum->num*2;default->0;};System.out.println("Result: "+result);}}

In this example, theswitch expression performs pattern matching. Different patterns are matched depending on the type of thevalue. If it’s aString, the length of the string is returned. If it’s anInteger, the value is doubled. Thedefault case is used if the value doesn’t match any of the specified patterns.

Note that pattern matching withinstanceof is more explicitly available from Java 17 onwards. But this will be a separate article.

Allowlisting/Blocklisting:

Allowing or disallowing specific characters or patterns in the input.

Allowlisting and blocklisting are often used in the context of security to control access to specific resources or actions. In Java, you can implement allowlisting or blocklisting using various approaches. Below is a simple example demonstrating how you might implement a basic allowlist and blocklist for a hypothetical resource.

Let’s say we have a class representing a resource and want to control access based on a list of allowed and disallowed entities.

xmlCopy

import java.util.ArrayList;import java.util.List;class ResourceAccessController { private List<String> whitelist; private List<String> blacklist; public ResourceAccessController() { whitelist = new ArrayList<>(); blacklist = new ArrayList<>(); } public void addToWhitelist(String entity) { whitelist.add(entity); } public void addToBlacklist(String entity) { blacklist.add(entity); } public boolean isAllowed(String entity) { // Check if the entity is in the whitelist and not in the blacklist return whitelist.contains(entity)&& !blacklist.contains(entity); }}public class Main { public static void main(String[] args) { ResourceAccessController accessController = new ResourceAccessController(); // Adding entities to the whitelist accessController.addToWhitelist("UserA"); accessController.addToWhitelist("UserB"); // Adding entities to the blacklist accessController.addToBlacklist("UserC"); // Checking access System.out.println("UserA allowed?: " + accessController.isAllowed("UserA")); // true System.out.println("UserB allowed?: " + accessController.isAllowed("UserB")); // true System.out.println("UserC allowed?: " + accessController.isAllowed("UserC")); // false System.out.println("UserD allowed?: " + accessController.isAllowed("UserD")); // false }}

In this example, theResourceAccessController class has methods to add entities to the allowlist and blocklist. TheisAllowed method checks if an entity is in the allowlist and not in the blocklist, thereby determining whether access is allowed. Note that this is a basic example, and in a real-world scenario, you might have more sophisticated access control mechanisms and data structures depending on your requirements.

Conclusion:

With “Input Validation”, you can ward off many attack vectors on a program and thus help make your software significantly more secure. The end user benefits from a more robust and reliable solution, and the software operator can also expect fewer unpleasant surprises from hackers. All in all, it is an approach that can be implemented anywhere in the application, does not require a lot of implementation effort, and, at the same time, is a cornerstone of security.

Happy Coding

]]>JavaSecure Coding PracticesSecurityhttps://svenruppert.com/posts/infection-method-sub-domain-takeover/Mon, 20 Nov 2023 14:37:29 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/infection-method-sub-domain-takeover/A subdomain takeover is a type of cybersecurity vulnerability that occurs when an attacker gains control of a subdomain of a website or a domain name. This attack can seriously affect the security and functionality of a web application or website. In this explanation, we’ll look at subdomain takeovers, how they work, the risks they pose, and how to prevent them.<![CDATA[

A subdomain takeover is a type of cybersecurity vulnerability that occurs when an attacker gains control of a subdomain of a website or a domain name. This attack can seriously affect the security and functionality of a web application or website. In this explanation, we’ll look at subdomain takeovers, how they work, the risks they pose, and how to prevent them.

Introduction to subdomains

To understand what a subdomain takeover is, we need to understand the concept of subdomains. In the context of the Internet, a domain name such as “example.com” serves as the primary address for a website. However, a domain name can be further divided into subdomains, which act as separate sections or branches of the main domain. Subdomains are often used for organisational purposes, categorising different website areas, or providing specific services.

For example, consider the subdomains of the abc.com domain:

• www.abc.com (for the main website)
• blog.abc.com (for a blog section)
• shop.abc.com (for an online shop)
• mail.abc.com (for email services)
• api.abc.com (for application programming interfaces)

These subdomains can be independently configured to provide different content or applications. Although subdomains are part of the same parent domain, they may be hosted on other servers or managed by different teams within the same organisation.

What is a subdomain takeover?

A subdomain takeover occurs when an unauthorised person or organisation gains control of a subdomain of a domain and thereby effectively takes ownership of it. This unauthorised control can lead to various security risks and potential misuse. The key factors contributing to subdomain takeover are misconfigurations, abandoned resources, and external service integrations.

Here are the main reasons why subdomain takeovers can occur:

https://youtu.be/rhrSC_4neAY

Misconfigurations:

Web administrators and developers can accidentally misconfigure DNS records or forget to remove DNS records for no more prolonged use subdomains. These misconfigurations can allow an attacker to take control of a subdomain.

Abandoned resources:

Organisations can create subdomains for specific projects, campaigns, or temporary services and then abandon them later. If an attacker discovers an abandoned subdomain, they can claim it as their own and potentially use it for malicious purposes.

External service integrations:

Many websites integrate third-party services, such as content delivery networks (CDNs), cloud services or hosted applications. These services often require the creation of subdomains that point to the third-party service provider’s infrastructure. If the external service becomes vulnerable, attackers can exploit it to take over the associated subdomain.

How subdomain takeovers work

Subdomain takeovers result from vulnerabilities due to misconfigured DNS records, abandoned resources, or compromised external services.

Incorrectly configured DNS entries:

CNAME Records: A common misconfiguration involves CNAME (canonical name) records. A CNAME record maps a subdomain to another domain or subdomain. If a subdomain is incorrectly pointed to a domain controlled by an attacker, the attacker can effectively take over the subdomain.

Wildcard entries:

Wildcard DNS records are used to direct all subdomains of a domain to a common destination. If a subdomain with a wildcard entry is no longer claimed or points to an attacker’s server, the attacker can control all subdomains.

Abandoned resources:

Subdomains created for temporary projects, campaigns, or services may be forgotten or abandoned after their purpose has been served. Attackers can search for and claim such subdomains if they are available for registration or reconfiguration. I want to describe an example I have found several times in projects.

Let’s assume a service is hosted externally. For example, a virtual machine is created and assigned a logical name in Heroku. This typically looks like this: You can choose a name to which the external operator’s domain is added. In this example, it isdemapp.heroku.com. This service is no longer needed and will be cancelled by the customer. However, there are still entries in the DNS configuration that point todemoapp.heroku.com. This can be, for example, a redirect from your own subdomain in this example,demoapp.abc.com. The attacker now creates an instance on Heroku and gives it the namedemoapp , leading to the complete namedemoapp.heroku.com. Now, the attacker only has to listen to the ports and analyse which requests arrive at this machine and can build his attack vector based on this. For example, he can set up a corresponding compromised service that responds to these requests.

Compromised external services:

Some subdomains are created to link to external services or resources provided by third parties. Attackers can tamper with the associated subdomains if these external services are compromised or misconfigured.

In these scenarios, the attacker effectively takes control of the subdomain by configuring the DNS records or exploiting vulnerabilities. Once they control the subdomain, they can host malicious content, launch phishing attacks, or commit other forms of cybercrime.

In these scenarios, the attacker effectively takes control of the subdomain by configuring the DNS records or exploiting vulnerabilities. Once they control the subdomain, they can host malicious content, launch phishing attacks, or commit other forms of cybercrime.

Risks and consequences of subdomain takeovers

Subdomain takeovers pose significant risks to websites’ and web applications’ security, reputation and functionality. The consequences of a successful subdomain takeover can be severe and far-reaching, including:

Data theft:

Attackers can use subdomains to steal sensitive data such as user credentials, personal information, or financial data.

Phishing attacks:

Subdomain takeovers are often used in phishing campaigns. Attackers can create convincing fake login pages or legitimate content to trick users into revealing their credentials.

Malware distribution:

Malicious files or software can be hosted on the compromised subdomain, which can then be used to distribute malware to unsuspecting visitors.

Denylisting:

Search engines and email providers can denylist the parent domain if they detect malicious activity on a subdomain. This may impact website visibility and email communications.

Brand damage:

A subdomain takeover can damage an organisation or website’s reputation and lose trust among users and customers.

Financial loss:

Depending on the severity of the attack, a company could face financial losses, lawsuits, and fines.

Loss of control:

Once an attacker has control of a subdomain, it becomes difficult for the rightful owner to regain control and limit the damage.

Legal consequences:

Unauthorised access to or control over subdomains can result in legal consequences and penalties for the attacker.

Practical examples

To illustrate the real-world impact of subdomain takeovers, let’s look at a few notable cases:

Uber:

In 2015, researchers discovered a subdomain takeover vulnerability in Uber’s system. The ride-hailing company had a subdomain called riders.uber.com that was vulnerable to a takeover. Attackers may have used this subdomain to launch phishing attacks against Uber users. Uber immediately addressed the issue after being informed by researchers.

GitHub pages:

GitHub Pages is a popular platform for hosting websites and documentation. In 2017, a researcher found that GitHub Page’s subdomains were vulnerable to a takeover when users deleted their repositories but left the associated DNS records intact. Attackers could then claim the subdomains and host malicious content. GitHub has since taken measures to prevent such takeovers.

Shopify:

Shopify, a central e-commerce platform, faced a subdomain adoption issue in 2019. An attacker discovered that he could claim abandoned Shopify store subdomains and potentially use them for fraud. Shopify has fixed the vulnerability and improved its security measures.

These cases show that subdomain takeovers can affect both small and large organisations. Timely identification and remediation of such vulnerabilities is critical to maintaining the security and trustworthiness of online services.

Prevent subdomain takeovers

Preventing subdomain takeovers requires a combination of proactive measures and best practices. Here are steps companies can take to mitigate the risk of subdomain takeovers:

Regular DNS Audits:

Perform routine DNS record checks to identify misconfigurations or abandoned subdomains. Remove unnecessary DNS records and update records for active subdomains.

Best practices for CNAME records:

When using CNAME records, be careful where they point. Make sure you trust the target and check CNAME configurations regularly.

Wildcard DNS records:

Use wildcard DNS records carefully. Avoid directing all subdomains to a single destination, as this can create a single point of failure.

Subdomain Registration Guidelines:

Implement clear guidelines for subdomain creation and management. Ensure that subdomains are registered with valid entities within the organisation and follow a consistent naming convention.

Automated scanning:

Use automated scanning tools to identify subdomains at risk of takeover. These tools can help identify misconfigurations and vulnerabilities.

Security of external services:

If you use external services requiring subdomains, ensure those services are secure. Regularly monitor and audit the security of these services to prevent potential vulnerabilities.

DNS rate limit:

Implement rate limiting for DNS requests to protect against reconnaissance attacks from potential attackers.

Access control:

Restrict access to DNS configuration and ensure only authorised personnel can change DNS records.

Vulnerability disclosure programs:

Encourage security researchers and users to report subdomain takeover vulnerabilities responsibly. Establish a process for receiving and processing such reports.

Monitor subdomain activity:

Monitor subdomain activity and set up alerts for suspicious or unauthorised changes to DNS records.

Conclusion

Subdomain takeovers represent a widespread cybersecurity threat and can significantly impact website owners and users. These takeovers are often due to misconfigured DNS records, abandoned resources, or compromised external services.

To protect against subdomain takeovers, companies/teams must proactively monitor and secure their subdomains. This includes regular DNS audits, best practices for managing DNS records, and a strong awareness of potential vulnerabilities in external services.

Happy Coding

Sven

]]>Securityhttps://svenruppert.com/posts/infection-method-domain-takeover/Fri, 10 Nov 2023 10:31:25 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/infection-method-domain-takeover/In this post, we will look at another method of infection. These are the attack vectors via domain names. This can happen at the main level, i.e. the domain itself, or via sub-domains. But what exactly is a domain takeover attack?<![CDATA[

In this post, we will look at another method of infection. These are the attack vectors via domain names. This can happen at the main level, i.e. the domain itself, or via sub-domains. But what exactly is a domain takeover attack?

A domain takeover is a cyberattack when an attacker gains control of a domain name owned by another person or organization. This can have severe consequences as the attacker can use the domain for malicious purposes, such as spreading malware, phishing, or taking control of a company’s online presence.

Below, we will look at different ways in which such a takeover can take place:

1. Expired domain :

A common possibility of domain takeover is that the owner of a domain forgets to renew it. When a domain registration expires, it may be available for purchase by anyone. Attackers can monitor the expiration of valuable domains and quickly register them as soon as they become available. If you have set up automatic renewal with your domain registrar, this should not happen. But sometimes companies (or people) get rid of old domains. This could be because the acquired company is now fully integrated and the domain is no longer needed, or because a project has expired, or or or. The problem, then, is that these domains may still be included in the internal configurations and firewall rules. The attacker then exploits the extended privileges typically intended for their own systems. If the old domains are now for sale, the attacker has purchased them regularly, and the own configurations still need to be updated, an attack can be carried out.

https://youtu.be/I5ZX5yX7rOI

2. Domain-Hijacking :

In some cases, attackers can use social engineering techniques to trick domain registrars or Domain Name System (DNS) providers into giving them domain control. This could mean impersonating the domain owner or providing false information to the domain registrar. Here, however, you have to think in different directions. Here is an example of the domain used for a reverse attack on a Maven repository. An open-source project named ABC does not have its associated domain secured. The Maven artefacts are stored in the central Maven Central, for example, under “org.abc”. The attacker now registered the domain abc.org and then contacted the operator of Maven Central, saying that he needed access to his Maven repository. He has lost his access to data. The operator of Maven-Central then asks the applicant to store a TXT entry with a specific code (in this case, it is the ticket number of the request) in the DNS configuration. Once this code is available via DNS, rights to the repository are granted. The attacker can now deposit his Maven artefacts in the repository and thus make them generally available.

3. DNS-Misconfiguration :

Sometimes, domain takeovers occur due to misconfigurations in DNS settings. Attackers can exploit these misconfigurations to gain control of the domain or subdomains and thereby redirect traffic to malicious servers.

3.1 What is a critical DNS misconfiguration?

A critical Domain Name System (DNS) misconfiguration is an error or error in the configuration of a DNS system that can have severe consequences for the availability, security, and functionality of a domain or network. DNS is a crucial Internet component that translates human-readable domain names (like example.com) into IP addresses that computers use to identify and communicate with each other. When DNS misconfiguration occurs, it can lead to various problems, including:

1. Service interruption :

A misconfigured DNS record can cause service outages, making a website or other online services inaccessible.

2. Vulnerabilities :

Misconfigurations can lead to security vulnerabilities, e.g. B., to reveal sensitive information, enable unauthorized access, or enable DNS-related attacks such as DNS cache poisoning or DNS spoofing.

3. Data Loss :

Incorrect DNS configurations can lead to data loss as changes to DNS records can result in email misdirection or loss of crucial domain-related information.

4. Performance Issues :

Suboptimal DNS configurations can slow down domain name resolution and cause delays in website loading or other network activity.

5. Traffic Diversion :

DNS misconfigurations can inadvertently direct traffic to the wrong IP addresses or locations, potentially leading to data leaks, man-in-the-middle attacks, or other unintended consequences.

6. Domain-Hijacking :

This is exactly the case of domain takeover considered here, in which unauthorized persons gain control of a domain and the services associated with it. Common examples of critical DNS misconfigurations include errors in DNS records (e.g. A, CNAME, MX, TXT records), incorrect IP address assignments, outdated or expired DNS information, and unauthorized changes to DNS settings .

4. Phishing/Theft Of Credentials :

Attackers can also use phishing attacks to trick domain owners or those with administrative access to domain management accounts into revealing their credentials. Once they have the credentials, they can log in and take control of the domain.

5. Subdomain-TakeOver :

A subdomain takeover occurs when an unauthorized person or organization gains control of a subdomain of a domain and thereby effectively takes ownership of it. This unauthorized control can lead to various security risks and potential misuse. The key factors contributing to subdomain takeover are misconfigurations, abandoned resources, and external service integrations.

6. DNS-Cache-Poisoning :

In some cases, attackers may attempt to poison DNS caches, causing them to resolve a legitimate domain into a malicious IP address. This can lead to a temporary domain takeover as users are redirected to the attacker’s server rather than the intended website.

Once the attacker gains control of a domain, they can use it for various malicious purposes, such as hosting fake websites for phishing attacks, distributing malware, or intercepting communications. This can damage the domain owner’s reputation, jeopardize user security and privacy, and result in legal and financial consequences for the legitimate domain owner.

An example from history:

A notable historical example of a domain takeover is the case of the Syrian Electronic Army (SEA) in 2013. The SEA was a group of hacktivists who supported the Syrian government and targeted various websites, social media accounts and domains to promote their political agenda. One of the most high-profile incidents involved the takeover of the Twitter accounts and the domain of several well-known media organizations.

In April 2013, SEA compromised the domain registration account of MarkMonitor, a domain registrar and provider of brand protection services. Using stolen credentials or other means, SEA accessed the New York Times’ domain registration records. They changed the domain’s DNS records, redirecting traffic from the New York Times website to a server controlled by the SEA. As a result, visitors to The New York Times website were greeted with a message from SEA instead of the expected news content.

This domain takeover disrupted the New York Times’ online operations and raised concerns about the security of domain registrars. The incident highlighted the importance of protecting domain management accounts and the potential impact of domain takeovers on well-known media organizations and their readership.

How do you protect yourself from these attacks?

To protect yourself from domain takeovers, it is essential for domain owners to:

1. Keep your domain registrations current and renew them on time.

2. Use strong authentication and authorization mechanisms for domain management accounts.

3. Regularly monitor and check your DNS configurations for misconfigurations.

4. Educate your team about the risks of social engineering and phishing attacks.

5. Use domain security services and technologies to detect and prevent unauthorized changes to domain settings.

This is not an exhaustive list, but it is a good start to preventing the most common attack vectors.

Conclusion:

We have seen many different attack vectors that can lead to a loss of control of the (sub)domain. These attacks are still prevalent and are successfully used even on large companies. Unfortunately, many small companies have exactly these vulnerabilities of abandoned resources in the sub-domain area. I recommend that every project member pay close attention to what else can be actively found in the configurations of the firewalls and DNS, even when it comes to testing and development infrastructure.

Happy Coding

Sven

]]>Securityhttps://svenruppert.com/posts/eclipsestore-high-performance-serializer/Mon, 09 Oct 2023 21:59:54 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/eclipsestore-high-performance-serializer/I will introduce you to the serializer from the EclipseStore project and show you how to use it to take advantage of a new type of serialization. Since I learned Java over 20 years ago, I wanted to have a simple solution to serialize Java-Object-Graphs, but without the serialization security and performance issues Java brought us. It should be doable like the following…<![CDATA[

I will introduce you to the serializer from the EclipseStore project and show you how to use it to take advantage of a new type of serialization.

Since I learned Java over 20 years ago, I wanted to have a simple solution to serialize Java-Object-Graphs, but without the serialization security and performance issues Java brought us. It should be doable like the following…

javaCopy

byte[]data=serializer.serialize(objectGraph);NodeobjectGraphDeserialized=serializer.deserialize(data);

If you want to know how this is doable with the new Open-Source Project EclipseSerializer? You are right here.

Before we look at the Open-Source project EclipseStore Serializer, I want to recap a bit of the challenges coming from the Java Serialization itself. This will be the background information to see how powerful the project is.

Java Serialization in a nutshell

Java Serialization is a mechanism provided by the Java programming language that allows you to convert the state of an object into a byte stream. This byte stream can be easily stored in a file, sent over a network, or otherwise persisted. Later, you can deserialize the byte stream to reconstruct the original object, effectively saving and restoring the object’s state.

Here are some key points about Java Serialization:

Serializable Interface: To make a Java class serializable, it needs to implement theSerializable interface. This interface doesn’t have any methods; it acts as a marker interface to indicate that the objects of this class can be serialized.

javaCopy

importjava.io.Serializable;publicclassMyClassimplementsSerializable{// class members and methods}

Serialization Process: To serialize an object, you typically useObjectOutputStream. You create an instance of this class and write your object to it. For example:

javaCopy

try(FileOutputStreamfileOut=newFileOutputStream("object.ser");ObjectOutputStreamout=newObjectOutputStream(fileOut)){MyClassobj=newMyClass();out.writeObject(obj);}catch(IOExceptione){e.printStackTrace();}

Deserialization Process: To deserialize an object, you useObjectInputStream. You read the byte stream from a file or network and then useObjectInputStream to recreate the object.

javaCopy

try(FileInputStreamfileIn=newFileInputStream("object.ser");ObjectInputStreamin=newObjectInputStream(fileIn)){MyClassobj=(MyClass)in.readObject();// Now 'obj' contains the deserialized object}catch(IOException|ClassNotFoundExceptione){e.printStackTrace();}

Versioning Considerations: If you change a serialised class’s structure (e.g., adding or removing fields or changing their types), the deserialization of old serialized objects may fail. Java provides mechanisms like serialVersionUID to help with versioning and compatibility.

Security Considerations: Serialization can be a security risk, especially if you deserialize data from untrusted sources. Malicious code can be executed during deserialization. To mitigate this risk, you should carefully validate and sanitize any data you deserialize or consider alternative serialization mechanisms like JSON or XML.

Custom Serialization: In your class, you can customize the serialization and deserialization process by providingwriteObject andreadObject methods. These methods allow you to control how the object’s state is written to and read from the byte stream.

In summary, Java Serialization is a valuable feature for persisting objects and sending them across a network. However, it comes with some challenges related to versioning and security, so it should be used cautiously, especially when dealing with untrusted data sources.

What are the security issues

Java Serialization can introduce several security issues, particularly when deserializing data from untrusted sources. Here are some of the security concerns associated with Java Serialization:

Remote Code Execution: One of the most significant security risks with Java Serialization is the potential for remote code execution. When you deserialize an object, the Java runtime system can execute arbitrary code contained within the serialized data. Attackers can exploit this to execute malicious code on the target system. This vulnerability can lead to serious security breaches.

Denial of Service (DoS): An attacker can create a serialized object with a large size, causing excessive memory consumption and potentially leading to a denial of service attack. Deserializing large objects can consume significant CPU and memory resources, slowing down or crashing the application.

Data Tampering: Serialized data can be tampered with during transmission or storage. Attackers can modify the serialized byte stream to alter the state of the deserialized object or introduce vulnerabilities.

Insecure Deserialization: Deserializing untrusted data without proper validation can lead to security issues. For example, if a class that performs sensitive operations is deserialized from untrusted input, an attacker can manipulate the object’s state to perform unauthorized actions.

Information Disclosure: When objects are serialized, sensitive information may be included in the serialized form. If this data is not adequately protected or encrypted, an attacker may gain access to sensitive information.

How To Mitigate Serialization Issues

To mitigate these security issues, consider the following best practices:

Avoid Deserializing Untrusted Data: If possible, avoid deserializing data from untrusted sources altogether. Instead, use safer data interchange formats like JSON or XML for untrusted data. (Or use the EclipseSerializer ;-) )

Implement Input Validation: When deserializing data, validate and sanitize the input to ensure it adheres to expected data structures and doesn’t contain unexpected or malicious data.

Use Security Managers: Java’s Security Manager can be used to restrict the permissions and actions of deserialized code. However, it’s important to note that Security Managers have been deprecated in newer versions of Java.

Whitelist Classes: Limit the classes that can be deserialized to a predefined set of trusted classes. This can help prevent the deserialization of arbitrary and potentially malicious classes.

Versioning and Compatibility: Be cautious when making changes to serialized classes. UseserialVersionUID to manage versioning and compatibility between different versions of serialized objects.

Security Libraries: Consider using third-party libraries like Apache Commons Collections or OWASP Java Serialization Security (Java-Serial-Killer) to help mitigate known vulnerabilities and prevent common attacks.

In summary, Java Serialization can introduce serious security risks, especially when dealing with untrusted data. It’s essential to take precautions, validate inputs, and consider alternative serialization methods or libraries to enhance security. Additionally, keeping your Java runtime environment up to date is crucial, as newer versions of Java may include security improvements and fixes for known vulnerabilities.

Why is JSON or XML not the perfect solution for the JVM?

Many papers and lectures recommend circumventing the security risks of serialization by using XML or JSON. This is a structured representation of the data that is to be transferred. There are also security problems, but I will address these in a separate article. However, what should be addressed are two things. First, the data must be converted into a text representation. This usually requires more data volume than with a pure binary model. In addition, data such as the binary data of images must be recoded so that only printable or UTF-8 characters can be transmitted. This process requires a lot of time and usually a lot of memory when transforming it into XML and back from XML into the original format.

The second point that causes problems in most cases is the data structure. In XML and JSON, object references can only be stored in a more manageable manner. This makes processing many times more complicated, slower and more resource-intensive. Even though many solid solutions can be used to convert Java objects into XML or JSON, I recommend looking for new approaches occasionally.

EclipseStore - Serializer - Practical Part

Now, let’s get to the practical stuff in this article. The dependency is needed first. To do this, we add the following instructions to the pom.xml. The first release was prepared when writing this article, and a SNAPSHOT version (1.0.0-SNAPSHOT) was available from the repositories. In this case, you still have to use the SNAPSHOT repositories (https://oss.sonatype.org/content/repositories/snapshots)

Definition inside the pom.xml.

xmlCopy

<dependency><groupId>org.eclipse.serializer</groupId><artifactId>serializer</artifactId><version>{maven-version-number}</version></dependency>

The rest will happen quickly once we’re ready and have fetched the dependency. For the first test, we created a class called Node. Each Node can have a right and a left child. With this, we can create a tree.

javaCopy

privateStringid;privateNodeleftNode;privateNoderightNode;

As an example, I created the following construct and then serialized and de-serialized it once using the serializer.

javaCopy

NoderootNode=newNode("rootNode");NodeleftChildLev01=newNode("Root-L");NoderightChildLev01=newNode("Root-R");leftChildLev01.addLeft(newNode("Root-L-R"));leftChildLev01.addLeft(newNode("Root-L-L"));rightChildLev01.addLeft(newNode("Root-R-L"));rightChildLev01.addRight(newNode("Root-R-R"));rootNode.addLeft(leftChildLev01);rootNode.addRight(rightChildLev01);Serializer<byte[]>serializer=Serializer.Bytes();byte[]data=serializer.serialize(rootNode);NoderootNodeDeserialized=serializer.deserialize(data);System.out.println(rootNode.toString());System.out.println(" ========== ");System.out.println(rootNodeDeserialized.toString());

Now, let’s see whether this also works with a Java object graph. To do this, the class representing the node is changed so that a father node can also be defined. Cycles can now be set up.

xmlCopy

public class GraphNode { private String id; private GraphNode parent; private List<GraphNode> childGraphNodes = new ArrayList<>();

We take the graph listed here as an example.

javaCopy

GraphNoderootNode=newGraphNode("rootNode");GraphNodechild01Lev01=newGraphNode("child01Lev01");GraphNodechild02Lev01=newGraphNode("child02Lev01");rootNode.addChildGraphNode(child01Lev01);rootNode.addChildGraphNode(child02Lev01);child01Lev01.setParent(rootNode);child02Lev01.setParent(rootNode);GraphNodechild01Lev02=newGraphNode("child01Lev02");GraphNodechild02Lev02=newGraphNode("child02Lev02");child01Lev01.addChildGraphNode(child01Lev02);child01Lev01.addChildGraphNode(child02Lev02);child01Lev02.setParent(child01Lev01);child01Lev02.setParent(child01Lev01);GraphNodechild01Lev03=newGraphNode("child01Lev03");GraphNodechild02Lev03=newGraphNode("child02Lev03");child01Lev03.setParent(child02Lev01);child02Lev03.setParent(child02Lev01);child02Lev01.addChildGraphNode(child01Lev03);child02Lev01.addChildGraphNode(child02Lev03);//creating cyclesrootNode.addChildGraphNode(child01Lev03);rootNode.setParent(child02Lev03);

This graph is also processed without any problems, without the cycles causing any issues.

javaCopy

Serializer<byte[]>serializer=Serializer.Bytes();byte[]data=serializer.serialize(rootNode);GraphNoderootNodeDeserialized=serializer.deserialize(data);

You can make the examples even more complex and try out the subtleties of inheritance. New data types from JDK17 are also supported. This means I have a potent tool to handle various tasks. For example, one use can be found in another Eclipse project called EclipseStore. A persistence mechanism is provided here based on this serialization. But your own small projects can also benefit from this. I will show how quickly this can be integrated into a residual service.

Building A Simple REST Service

If you want to create a simple REST service for transferring byte streams in Java without using Spring Boot, you can use the Java SE API and the HttpServer class from the com.sun.net.httpserver package, which allows you to create an HTTP server.

• We create an HTTP server on port 8080 and define a context for handling requests to “/api/bytestream.”
• The ByteStreamHandler class handles both POST requests for uploading byte streams and GET requests for downloading byte streams.
• For POST requests, it reads the incoming byte stream, processes it as needed, and sends a response.
• For GET requests, it sends a predefined byte stream as a response.

Remember that this is a simple example, and you can expand upon it to handle more complex use cases and error handling as needed for your specific application. Also, note that the com.sun.net.httpserver package is part of the JDK, but it may not be available in all Java distributions.

javaCopy

publicclassByteStreamHandlerimplementsHttpHandler{@Overridepublicvoidhandle(HttpExchangeexchange)throwsIOException{StringrequestMethod=exchange.getRequestMethod();if(requestMethod.equalsIgnoreCase("POST")){// Handle POST requests for uploading byte streamshandleUpload(exchange);}elseif(requestMethod.equalsIgnoreCase("GET")){// Handle GET requests for downloading byte streamshandleDownload(exchange);}}privatevoidhandleUpload(HttpExchangeexchange)throwsIOException{// Get the input stream from the requestInputStreaminputStream=exchange.getRequestBody();// Read the byte stream and process it as neededByteArrayOutputStreambyteArrayOutputStream=newByteArrayOutputStream();byte[]buffer=newbyte[1024];intbytesRead;while((bytesRead=inputStream.read(buffer))!=-1){byteArrayOutputStream.write(buffer,0,bytesRead);}// Process the uploaded byte stream// (e.g., save to a file or perform other actions)byte[]data=byteArrayOutputStream.toByteArray();// Do something with 'data'Serializer<byte[]>serializer=Serializer.Bytes();GraphNodedeserialized=serializer.deserialize(data);//process the dataSystem.out.println("deserialized = "+deserialized);// Send a response (you can customize this)Stringresponse="Byte stream uploaded successfully.";exchange.sendResponseHeaders(200,response.length());OutputStreamos=exchange.getResponseBody();os.write(response.getBytes());os.close();}privatevoidhandleDownload(HttpExchangeexchange)throwsIOException{// Simulate generating and sending a byte stream as a responseStringresponse="Hello, Byte Stream!";Serializer<byte[]>serializer=Serializer.Bytes();byte[]data=serializer.serialize(response.getBytes());exchange.sendResponseHeaders(200,data.length);OutputStreamos=exchange.getResponseBody();os.write(data);os.close();}}

Conclusion:

We’ve looked at the typical problems with Java’s original serialization and how cumbersome the implementation is to use. The detour via JSON and XML is unnecessary when communicating from JVM to JVM using the open-source Eclipse Serializer project. There are no restrictions when it comes to modelling a graph, as not only are the current new data types up to and including JDK17 already processed, but cycles within the graph are also no problem.

Using the Serializable interface is also unnecessary and does not influence processing. The easy handling allows it to be used even in tiny projects such as the REST service shown here using on-board JDK resources. A larger project that uses the serializer is the open-source project EclipseStore. A high-performance persistence mechanism for the JVM is offered here.

Happy Coding

Sven

]]>EclipseStoreJavaSecurityhttps://svenruppert.com/posts/eclipsestore-storing-more-complex-data-structures/Fri, 06 Oct 2023 12:22:50 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/eclipsestore-storing-more-complex-data-structures/How to store complex data structures using EclipseStore? Are there any restrictions? How can you work more efficiently on such structures? We will now get to the bottom of these questions here.<![CDATA[

How to store complex data structures using EclipseStore? Are there any restrictions? How can you work more efficiently on such structures? We will now get to the bottom of these questions here.

In the first part of my series, I showed how to prepare EclipseStore for use in a project. We also initialized the StorageManager and saved, modified and deleted the first data. But what about more complex structures? Can you use inheritance? To do this, we will now create a small class model.

First, let’s look at what inheritance looks like. To do this, we take an interface calledBaseInterfaceA , an implementationBaseClassA and a derivativeLevelOneA. We will now try to save this and see how it behaves depending on the input when saving.

Here is the listing of the classes and interfaces involved. Since we already saw in the first part that the implementation variant has no influence, I will use a standard implementation regarding getters and setters or the constructors.

javaCopy

publicinterfaceBaseInterfaceA{   publicdefaultStringallUpperCase(Stringvalue){       returnvalue.toUpperCase();   }}publicclassBaseClassAimplementsBaseInterfaceA{   privateStringvalueBaseA;   @Override   publicStringtoString(){       return"BaseClassA{"+               "valueBaseA='"+valueBaseA+'\''+               '}';   }   @Override   publicbooleanequals(Objecto){       if(this==o)returntrue;       if(o==null||getClass()!=o.getClass())returnfalse;       BaseClassAthat=(BaseClassA)o;       returnObjects.equals(valueBaseA,that.valueBaseA);   }   @Override   publicinthashCode(){       returnObjects.hash(valueBaseA);   }   publicStringgetValueBaseA(){       returnvalueBaseA;   }   publicvoidsetValueBaseA(StringvalueBaseA){       this.valueBaseA=valueBaseA;   }}publicclassLevelOneAextendsBaseClassA{   privateStringvalueOneA;   publicLevelOneA(StringvalueOneA){       this.valueOneA=valueOneA;   }   @Override   publicbooleanequals(Objecto){       if(this==o)returntrue;       if(o==null||getClass()!=o.getClass())returnfalse;       if(!super.equals(o))returnfalse;       LevelOneAlevelOneA=(LevelOneA)o;       returnObjects.equals(valueOneA,levelOneA.valueOneA);   }   @Override   publicinthashCode(){       returnObjects.hash(super.hashCode(),valueOneA);   }   publicStringgetValueOneA(){       returnvalueOneA;   }   publicvoidsetValueOneA(StringvalueOneA){       this.valueOneA=valueOneA;   }   @Override   publicStringtoString(){       return"LevelOneA{"+               "valueOneA='"+valueOneA+'\''+               '}';   }}

Case I: Direct saving of the respective classes

The entities are packed directly into a list and saved in the first case. As we can see, there are no special features here. This is the same case as the first article. The output of theprintElements method is exactly what we expected.

xmlCopy

public static void main(String[] args) {   EmbeddedStorageManager storageManager = EmbeddedStorage.start();   storageManager.setRoot(new ArrayList<LevelOneA>());   storageManager.storeRoot();   List<LevelOneA> root = (List<LevelOneA>) storageManager.root();   root.add(new LevelOneA("levelOneA - 01"));   root.add(new LevelOneA("levelOneA - 02"));   root.add(new LevelOneA("levelOneA - 03"));   storageManager.storeRoot();   storageManager.shutdown();   storageManager = EmbeddedStorage.start();   List<LevelOneA> rootLoaded = (List<LevelOneA>) storageManager.root();   rootLoaded.forEach(System.out::println);   System.out.println(" ========== ");   storageManager.shutdown();}

console output:

LevelOneA{valueOneA=‘levelOneA - 01’}

LevelOneA{valueOneA=‘levelOneA - 02’}

LevelOneA{valueOneA=‘levelOneA - 03’}

==========

Case II: Save as a base class

Now, let’s consider the case where a class is passed to the StorageManager for storage as one of its base classes. In this case, theLevelOneA class is passed as the base classBaseClassA. It should be noted that this primary type was also used when defining the root list.

xmlCopy

EmbeddedStorageManager storageManager = EmbeddedStorage.start();storageManager.setRoot(new ArrayList<BaseClassA>());storageManager.storeRoot();List<BaseClassA> root = (List<BaseClassA>) storageManager.root();root.add(new LevelOneA("levelOneA - 01"));root.add(new LevelOneA("levelOneA - 02"));root.add(new LevelOneA("levelOneA - 03"));storageManager.storeRoot();storageManager.shutdown();storageManager = EmbeddedStorage.start();List<BaseClassA> rootLoaded = (List<BaseClassA>) storageManager.root();rootLoaded.forEach(System.out::println);System.out.println(" ========== ");storageManager.shutdown();

The output again meets our expectations.

LevelOneA{valueOneA=‘levelOneA - 01’}

LevelOneA{valueOneA=‘levelOneA - 02’}

LevelOneA{valueOneA=‘levelOneA - 03’}

==========

Case II: Saving an implementation as an interface

Now let’s move on to the case in which we proceed via the interface. There are no problems here either. Everything behaves as expected.

xmlCopy

EmbeddedStorageManager storageManager = EmbeddedStorage.start();storageManager.setRoot(new ArrayList<BaseInterfaceA>());storageManager.storeRoot();List<BaseInterfaceA> root = (List<BaseInterfaceA>) storageManager.root();root.add(new LevelOneA("levelOneA - 01"));root.add(new LevelOneA("levelOneA - 02"));root.add(new LevelOneA("levelOneA - 03"));storageManager.storeRoot();storageManager.shutdown();storageManager = EmbeddedStorage.start();List<BaseInterfaceA> rootLoaded = (List<BaseInterfaceA>) storageManager.root();rootLoaded.forEach(System.out::println);System.out.println(" ========== ");storageManager.shutdown();

The output on the console is still unchanged.

LevelOneA{valueOneA=‘levelOneA - 01’}

LevelOneA{valueOneA=‘levelOneA - 02’}

LevelOneA{valueOneA=‘levelOneA - 03’}

==========

Case IV: Saving a mixed list via a common interface

Now if we define a list of typeBaseInterfaceA as root and then add instances of different implementations. How does EclipseStore behave during the save process? To make a long story short, it works as intended. All instances are stored neatly after deployment. So we have no loss of information.

The console output will then look like this.

LevelOneA{valueOneA=‘levelOneA - 01’}

BaseClassA{valueBaseA=‘BaseClassA - 02’}

LevelOneA{valueOneA=‘levelOneA - 03’}

==========

Storing lists, trees and graphs

Now that we’ve looked at whether inheritance is supported as much as we hoped, we’re getting to the point where we can start thinking about the data structures themselves. We have seen that simple entities and lists of entities are relatively easy for EclipseStore. Now, let’s go one step further and look at trees and graphs.

Storing of trees

In computer science, we understand a tree to be a data structure that can have branches but does not contain any cycles. This makes them a special form of graphs, just as lists are a special form of trees. So, let’s come to a tree implementation in which a node can have two child nodes. Of course, you can also build this structure with n child nodes, but this will not bring any added value in our case. If we create such a tree and give the root node to the StorageManager, we can save this tree without any further action. Modifying individual elements also works as usual. You can also change the desired segment and save on this or any higher-level piece.

javaCopy

EmbeddedStorageManagerstorageManager=EmbeddedStorage.start();NoderootNode=newNode("rootNode");storageManager.setRoot(rootNode);storageManager.storeRoot();NodeleftChildLev01=newNode("Root-L");NoderightChildLev01=newNode("Root-R");leftChildLev01.addLeft(newNode("Root-L-R"));leftChildLev01.addLeft(newNode("Root-L-L"));rightChildLev01.addLeft(newNode("Root-R-L"));rightChildLev01.addRight(newNode("Root-R-R"));rootNode.addLeft(leftChildLev01);rootNode.addRight(rightChildLev01);storageManager.storeRoot();storageManager.shutdown();storageManager=EmbeddedStorage.start();NoderootLoaded=(Node)storageManager.root();System.out.println(rootLoaded.toString());System.out.println(" ========== ");storageManager.shutdown();

Output to the console (subsequently formatted for readability):

Node{id=‘rootNode’,

leftNode=Node{id=‘Root-L’,

leftNode=Node{id=‘Root-L-L’,

leftNode=null,

rightNode=null

},

rightNode=null

},

rightNode=Node{id=‘Root-R’,

leftNode=Node{id=‘Root-R-L’,

leftNode=null,

rightNode=null

},

rightNode=Node{id=‘Root-R-R’,

leftNode=null,

rightNode=null

}

}

}

Storing graphs

Unfortunately, you don’t just have to deal with lists and trees. Complex data models often have cycles. This can come about, for example, through bidirectional relationships. Unwanted cycles can also arise in a model if it grows over time and is repeatedly expanded. Whether these cycles are tightly or loosely coupled plays a minor role. The following example creates a chart that contains multiple cycles. Can the diagram then be saved, modified and loaded? Does EclipseStore recognize these structures and can resolve or break the loops?

In this example, the graph consists of nodes of theGraphNode class. This class contains an attribute of type String to store an ID. There is also a reference to the parent node and a list of child nodes. This means you can now create any nested chart you want.

In this case, the graphic looks like this.

javaCopy

GraphNoderootNode=newGraphNode("rootNode");GraphNodechild01Lev01=newGraphNode("child01Lev01");GraphNodechild02Lev01=newGraphNode("child02Lev01");rootNode.addChildGraphNode(child01Lev01);rootNode.addChildGraphNode(child02Lev01);child01Lev01.setParent(rootNode);child02Lev01.setParent(rootNode);GraphNodechild01Lev02=newGraphNode("child01Lev02");GraphNodechild02Lev02=newGraphNode("child02Lev02");child01Lev01.addChildGraphNode(child01Lev02);child01Lev01.addChildGraphNode(child02Lev02);child01Lev02.setParent(child01Lev01);child01Lev02.setParent(child01Lev01);GraphNodechild01Lev03=newGraphNode("child01Lev03");GraphNodechild02Lev03=newGraphNode("child02Lev03");child01Lev03.setParent(child02Lev01);child02Lev03.setParent(child02Lev01);child02Lev01.addChildGraphNode(child01Lev03);child02Lev01.addChildGraphNode(child02Lev03);//creating cyclesrootNode.addChildGraphNode(child01Lev03);rootNode.setParent(child02Lev03);EmbeddedStorageManagerstorageManager=EmbeddedStorage.start();storageManager.setRoot(rootNode);storageManager.storeRoot();

The node rootNode is passed to the StorageManager as root and saved. A subsequent loading of the root results in the previously created graph.

As we can see from this example, EclipseStore can store graphs without any problems. Cycles are acceptable here.

Conclusion:

We have now seen that EclipseStore can store any data structure in its entirety; lists, trees and graphs are not a limitation. This allows us to create the data model independently of the persistence layer. This is a possibility that I have always been looking for in the last 20 years of my Java activities.

We will now deal with the intricacies of EclipseStore in the following parts. It remains exciting.

Happy coding

Sven

]]>EclipseStoreJavahttps://svenruppert.com/posts/how-to-start-with-eclipsestore-01/Wed, 04 Oct 2023 10:00:04 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/how-to-start-with-eclipsestore-01/We will take the first steps with the Eclipse Store here. I will show what the Eclipse Store is, how you can integrate it into your project and what the first steps look like from a developer’s perspective. All in all, it is a practical introduction.<![CDATA[

We will take the first steps with the Eclipse Store here. I will show what the Eclipse Store is, how you can integrate it into your project and what the first steps look like from a developer’s perspective. All in all, it is a practical introduction.

What is Eclipse Store?

First, I would like to briefly explain what Eclipse Store actually is. Simply put, this is a mechanism for storing Java object trees. This initially sounds very theoretical, but it is much simpler than you might think.

When I started developing Java applications in 1996, I dreamed of being able to easily store the objects I created in my application. I found the assumption that serialization could be used to write everything to the hard drive as a byte stream very good. Unfortunately, the reality looked different or still looks different today. You were quickly confronted with various technologies, all with their own characteristics. I still remember very clearly the first steps in which the application’s data had to be persisted via a JDBC interface. Who doesn’t know the JDBC-ODBC interface from back then? A typical application now had JDBC connections, mapping layers that implement everything in SQL and, of course, caches in various places so that the application’s responsiveness remained tolerable for the user. It’s not easy when dealing with a more complex data model.

With Eclipse Store, we now have a tool that eliminates most of these technology layers. So we have finally achieved what was promised to us in 1996.

Let’s now come to the implementation of our own project. For this, we need the maven dependencies. For this example, I used the first final version. This is based on Microstream Version 8 and is the basis for the Eclipse project. As far as I know, Microstream is not being actively developed further.

xmlCopy

<dependencies><dependency><groupId>org.eclipse.store</groupId><artifactId>storage-embedded</artifactId><version>{maven-version}</version></dependency></dependencies>

Now that we have added the necessary dependencies to the project, we can start with the first lines of source code.

Basic first Steps:

The Storage Manager:

We will now look at the essential elements of architecture. The linchpin of the persistence solution is the “Storage Manager”. This is the interface through which the behaviour can be configured. All entities are also transferred to persistence, retrieved and deleted. The storage manager itself is available in various forms. We will focus on embedded storage here for now. This implementation is an in-memory solution that accesses the local hard drive directly. You shouldn’t let the name confuse you. As is usual with RDBMS systems, it is not a tiny solution that allows you to do a little testing. The implementation of embedded storage is absolutely suitable for production.

So, an instance of this implementation is needed. The easiest way to get this is to call:

javaCopy

**final****EmbeddedStorageManagerstorageManager=EmbeddedStorage.start();**

The chosen implementation depends on which dependencies have been defined in the pom.xml. In our case, it’s the embedded implementation. We will discuss various other implementations in the following parts. We will also take a closer look at how the behaviour can be configured later. These parameters are irrelevant for the first steps and can safely be ignored.

Storage Root:

The storage manager now needs to know the root of the graphs to be stored. You can imagine this as described below.

If we only have one element that needs to be stored, then no further wrapper is necessary. However, if we have two parts that we want to save, we need a container to hold both elements. This container is then the root through which initial access to all elements can occur. This can be a list or a class specifically tailored to your needs. This container must now be made available for persistence.

Let’s look at an example. Two different strings should be stored here. A list is used as a container. This means that the instance of the list is the StorageRoot element. This must now be declared as an anchor point.

javaCopy

EmbeddedStorageManagerstorageManager=EmbeddedStorage.start();storageManager.setRoot(newArrayList<>());storageManager.storeRoot();

The “storeRoot()” instruction causes this modification to be saved at the root. We will see this “storeRoot()” method call more often. This persists in all changes in the graph, starting from the root. In our case, the root itself is saved.

How can another element be added to this container? There is no need to keep a reference to this container. Instead, we can ask the Storage Manager for this reference. What should not be forgotten at this point? If you overwrite the root, all previously saved elements will be deleted. These are then no longer under the supervision of persistence.

xmlCopy

**List<String> root = (List<String>) storageManager.root();**

A downer at this point. Unfortunately, the “root()” method is not typed. Here, you have to know exactly what type the container is. We’ll look at how to deal with this better. I ask for patience here.

Any number of elements can now be added or removed from this list. To save the changes, the StorageManager is informed of this. Let’s look at this in a little more detail. We will expand and modify the previous source code for this. Instead of simple strings, a separate class with two attributes is now used. The first attribute is again a string and represents the data value. The second attribute is an instance of the type LocaDateTime and is an automatically set timestamp. We can already see here that it is no longer a simple attribute. The constructor takes over the data value, which can be modified later. The timestamp cannot be set explicitly. There is only one getter here. Such constructs can also be processed by EclipseStore without any further action.

javaCopy

publicclassDataElement{privateStringvalue;privateLocalDateTimetimestamp;publicDataElement(Stringvalue){this.value=value;this.timestamp=LocalDateTime.now();}publicvoidsetValue(Stringvalue){this.value=value;this.timestamp=LocalDateTime.now();}publicStringgetValue(){returnvalue;}publicLocalDateTimegetTimestamp(){returntimestamp;}@OverridepublicStringtoString(){return"DataElement{"+"value='"+value+'\''+", timestamp="+timestamp+'}';}

Case I - We add elements to the container:

In our case, the simplest case is to add one or more elements to the root element, a list of type DataElement. This is done by adding the desired elements to the list and calling the “storeRoot()” method at the root level or saving the elements yourself.

xmlCopy

EmbeddedStorageManager storageManager = EmbeddedStorage.start(); storageManager.setRoot(new ArrayList<DataElement>()); storageManager.storeRoot(); List<DataElement> root = (List<DataElement>) storageManager.root(); root.add(new DataElement("Nr 1")); root.add(new DataElement("Nr 2")); storageManager.storeRoot();

And here is the method in which the elements are specified directly.

javaCopy

DataElementd3=newDataElement("Nr 3");DataElementd4=newDataElement("Nr 4");root.add(d3);root.add(d4);storageManager.storeAll(d3,d4);

Case II - We remove elements from the container:

The opposite operation of adding elements is deleting them. Here, too, the process is very straightforward. However, looking at the different options is unusual at the beginning. First, it’s the most intuitive version. Here, the element is removed from the holding object, the root itself. This reduced instance is then passed on to the StorageManager with the request to save this modification.

javaCopy

root.remove(d3);storageManager.store(root);

However, you can also use the removed element yourself. For this purpose, the element previously removed from the holding instance is passed to the StorageManager for storage. This approach works but is anything but intuitive. At this point, for reasons of comprehensibility, you should not write it. Your colleagues will be grateful for that.

javaCopy

root.remove(d4);storageManager.store(d4);

Case III - We modify the elements within the container:

Since elements can now be added and removed, the only thing missing is the modification. Here, too, the procedure is analogous to adding. The instance is modified and then passed to the StorageManager for storage. The saving process can, of course, also take place via a holding element. For readability reasons, however, saving the modified elements yourself is strongly advisable. Exception can, of course, be when you change a lot of elements within a holding instance.

javaCopy

d3.setValue("Nr 3. modified");storageManager.store(d3);

Fazit:

We have now seen how easy it is to get started with EclipseStore. Elements can now be saved without any further effort. This way, you can realize the first small applications. But that’s not the end of it. The following parts will address the intricacies and challenges of more complex applications. So it remains exciting.

Happy Coding

Sven

]]>EclipseStoreJavahttps://svenruppert.com/posts/tdd-and-the-impact-on-security/Wed, 28 Jun 2023 16:08:03 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/tdd-and-the-impact-on-security/Test-driven development (TDD) is a software development approach that prioritizes writing automated tests while creating the actual code. There follows a cycle of writing a failed test, writing the code to make the test pass, and then refactoring the code. TDD was originally developed to ensure the quality, maintainability and expandability of the software created over the long term. The specific knowledge about the individual source text passages should also be shown in the tests. Thus, a transfer of responsibility between developers is supported. Better than any documentation, tests are always up-to-date regarding the function that has been implemented in the source code.<![CDATA[

Test-driven development (TDD) is a software development approach that prioritizes writing automated tests while creating the actual code. There follows a cycle of writing a failed test, writing the code to make the test pass, and then refactoring the code. TDD was originally developed to ensure the quality, maintainability and expandability of the software created over the long term. The specific knowledge about the individual source text passages should also be shown in the tests. Thus, a transfer of responsibility between developers is supported. Better than any documentation, tests are always up-to-date regarding the function that has been implemented in the source code.

However, TDD also has a positive impact on the security of a program. And that’s what we’re going to look at now.

But first, let’s get back to TDD. There are a variety of theoretical approaches here. I will briefly touch on them here to give an overview. The question to be addressed is whether the TDD approach influences the effect.

https://youtu.be/S-GKA6KJwPc

Classic TDD - Kent Beck

Unit tests and units tested with them are continuously developed in parallel. The actual programming is done in small, repeating micro-iterations. One such iteration, which should only take a few minutes, consists of three main parts, known as the red-green refactoring.

Red: Write a test to test a new behaviour (functionality) to be programmed. You start with the simplest example. If the feature is older, it could also be a known bug or new functionality to be implemented. This test is initially not fulfilled by the existing program code, so it must fail.

Green: Change the program code with as little effort as possible and add to it until it passes all tests after the test run then initiated.

Then clean up the code (refactoring): remove repetitions (code duplication), abstract if necessary, align with the binding code conventions, etc. In this phase, no new behaviour may be introduced that would not already be covered by tests. After each change, the tests are performed; if they fail, it is impossible to replicate what appears to be a bug in the code being used. Cleanup aims to make the code simple and easy to understand.

These three steps are repeated until the known bugs are fixed, the code provides the desired functionality, and the developer can’t think of any more useful tests that could fail. The program-technical unit (unit) treated in this way is considered complete for now. However, the tests created together with her are retained to be able to test again after future changes as to whether the behavioural aspects that have already been achieved are still fulfilled.

For the changes in step 2 - also called transformations - to lead to the goal, each change must lead to a more general solution; for example, she may not only edit the current test case at the expense of others. Tests that get more and more detailed bring the code to a more and more general solution. Regular attention to transformation priorities leads to more efficient algorithms.

Consistently following this approach is an evolutionary design method, where every change evolves the system.

Outside in TDD

Outside-In Test-Driven Development (TDD) is an approach to software development that places emphasis on starting the development process first by creating high-level acceptance tests or end-to-end tests that demonstrate the desired behaviour of the system from his point of view to define users or external interfaces. It is also commonly referred to as behaviour-directed development (BDD).

With Outside-In TDD, the development process begins with writing a failed acceptance test that describes the desired behaviour of the system. This test is usually written from a user’s perspective or a high-level component that interacts with the system. The test is expected to initially fail as the system does not have the required functionality.

Once the first acceptance test has been performed, the next step is to write a failing unit test for the smallest possible unit of code that will pass the acceptance test. This unit test defines the desired behaviour of a specific module or component within the system. The unit test fails because the corresponding code still needs to be implemented.

The next step is implementing the code to make the failed unit test succeed. The code is written incrementally, focusing on the immediate needs of the failed test. The developer writes further tests and code step by step until the acceptance test is passed and the desired behaviour of the system is achieved.

The idea behind Outside-In TDD is to drive the development process from the outside, starting with the higher-level behaviour and moving inward to the lower-level implementation details. This approach helps ensure that the system is developed to meet the needs and expectations of its users. It also encourages component testability and decoupling by encouraging the creation of small, focused tests and modular code.

By practising outside-in TDD, developers can gain confidence in their code and ensure the system behaves as expected. It also helps uncover design flaws early in the development process and encourages the creation of loosely coupled and easily maintainable code.

Overall, Outside-In TDD is a methodology that combines testing and development, focusing on providing value to users and driving the development process from an outside perspective.

Acceptance Driven TDD

In test-driven development, system tests are continuously developed or at least specified before the system. In test-driven development, system development is no longer to meet written requirements but to pass specified system tests.

Acceptance Test-Driven Development (ATDD), while related to Test-Driven Development, differs in approach from Test-Driven Development. Acceptance test-driven development is a communication tool between the customer or user, the developer and the tester to ensure the requirements are well described. Acceptance test-driven development does not require automation of test cases, although it would be useful for regression testing. The acceptance tests for test-driven development must also be legible for non-developers. In many cases, the tests of test-driven development can be derived from the tests of acceptance test-driven development. For this approach to be used for system security, the boundary conditions must go beyond the normal technical aspects, which are only considered in some cases.

What test coverage should I use?

It is important for the effect on security that tests run automatically. These then lead to test coverage that can be measured and compared to previous runs. The question arises as to which test coverage will be most suitable. It has been shown that there are significant differences in the strength of the respective approaches. I am a strong proponent of mutation test coverage, as it is one of the most effective test coverages. If you want to know more here, I refer to my video on “Mutation Testing in German” or “Mutation Testing in English” on my YouTube channel. In any case, it is important that the better the test coverage, the greater the effect that can be achieved.

How exactly is the security of the application supported?

1. Identify vulnerabilities early: Writing tests before implementing features help identify potential security gaps early. By considering security concerns during the test design phase, developers can anticipate possible attack vectors and design their code with security guidelines in mind. This proactive approach makes it possible to detect security problems before they become deeply embedded in the code base.

2. Encouraging Safe Coding Practices: TDD encourages writing modular, well-structured, and testable code. This enables developers to adopt secure coding practices such as input validation, proper error handling, and secure data storage. By writing tests that simulate different security scenarios, developers are more likely to consider edge cases and verify that their code behaves securely.

3. Regression Testing for Security: TDD relies on running automated tests regularly to maintain existing functionality as new code is added. This approach helps prevent a decline in security-related issues. If a security-related test case exists and initially passes, subsequent changes to it can be tested to ensure that vulnerabilities were not accidentally introduced.

4. Building a Security-Aware Culture: TDD’s incorporation of security testing as an integral part of the development process helps foster a security-aware culture within the development team. By performing security-focused testing alongside functional testing, developers are more likely to prioritize security and see it as a fundamental aspect of their job. This mindset encourages a proactive attitude towards security rather than just seeing it as a side issue.

5. Facilitate Collaboration and Code Review: TDD encourages collaboration and code review between team members. When developers first write tests, they can discuss security concerns and get feedback from peers. This collaborative approach increases overall security awareness within the team and enables early detection and resolution of security issues.

Although TDD is not a security panacea, it offers a systematic and disciplined development approach that indirectly strengthens software security posture. By promoting security-centric thinking, promoting secure coding practices, and facilitating early detection of vulnerabilities, TDD can help build more secure and resilient applications. However, it is important to note that additional security measures such as threat modelling, code analysis, and penetration testing should also be included to meet security requirements fully.

A few more practical approaches

What are Compliance Issues, and how to remove them?

The compliance issues are elements that run under a license and may not be used in this way in the project. This can happen in all technology areas within a project. Libraries or frameworks used do not usually change their license over the years, but it occasionally happens. All dependencies of the elements used must also be checked. And that’s up to the last link.

One must replace the affected element with a semantic equivalent available under an appropriate license to fix such a violation.

What are Vulnerabilities, and how to remove them?

Vulnerabilities are unintentional errors in the source code that allow the system to be manipulated to its advantage. These errors can occur throughout the tech stack. It is important to recognize all vulnerabilities in all technologies used to get an overview of which vulnerabilities can be exploited in which combination for an attack on this system. Individual considerations need to be clarified at this point. This holistic view enables the recognition of the different attack vectors and the identification of weak points that cannot be exploited in this system. This allows you to use your efforts in a targeted manner to secure the system.

Why do we need efficient dependency management?

What tools are available to developers to address compliance issues and vulnerabilities? The answer is obvious. This is where very effective dependency management combined with robust test coverage comes in handy. Because, in all cases, dependencies have to be exchanged. Even though it’s a surrogate for compliance issues, fixing vulnerabilities is the same dependency in a different version. This can be a higher or lower version. Overall, it is about the clever combination of the various elements. Strong test coverage facilitates the subsequent test to determine whether the functionality is still available.

Conclusion:

In summary, strong test coverage is beneficial when modifying and exchanging external elements as well as your own source code. Here you can rely on the results of the CI environment without having to add manual verifications and thus implement a fast and lean release process with as little time loss as possible.

Quality and safety support each other.

Happy coding

Sven

]]>SecurityTDDhttps://svenruppert.com/posts/introduction-to-the-linux-foundations-slsa-project/Sat, 10 Dec 2022 21:56:43 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/introduction-to-the-linux-foundations-slsa-project/Supply Chain Security is a hot topic these days. And more and more, we as developers are dealing with this daily. But what does this mean for us, and how is this influencing our job? I want to give an overview of common attacks against the Software Supply Chain from the developer’s view and will introduce the Open Source project SLSA from the Linux Foundation.<![CDATA[

Supply Chain Security is a hot topic these days. And more and more, we as developers are dealing with this daily. But what does this mean for us, and how is this influencing our job? I want to give an overview of common attacks against the Software Supply Chain from the developer’s view and will introduce the Open Source project SLSA from the Linux Foundation.

a) Who is the project SLSA

Various experts from the security field started this project to share their knowledge, leading to the mentioned project. There is no company or governmental organisation behind this project. It is a pure Open Source project under the umbrella of the Linux Foundation.

b) What is the goal of this project?

The SLSA project is open-source and will not provide its own source code. It is, therefore, not a classic open-source project to publish a specific solution. Instead, it is a documentation project to process knowledge about supply chain security in software development and make it freely accessible. The structure contains examples of where the events or attacks mentioned have been successfully taken and which countermeasures should be taken. We’ll look at the security levels provided a little later. The aim here is to allow the reader to prepare for these threats with specific steps slowly.

After all, based on your options and the current environment, you have to decide which next steps and measures make sense to implement.

https://youtu.be/r8moPP8EACc

c) What is the current status of the project?

The project is currently (beginning of 2022) still in the alpha phase. There is a lot of documentation, but some places still provide references to future content. So it’s always worth taking a look inside. Since this is an open-source project, you can, of course, also contribute yourself. Here you have the opportunity to make your expertise available to others.

SLSA security levels?

a) Why do we need these security levels?

When you look around the SLSA project website for the first time, the description of the security levels quickly catches your eye. But what are these levels all about? With these levels, you want to achieve two things. First, it should help to enable a self-assessment. This first consideration serves to classify the own existing environment and to identify the primary weak points. The second goal of this level is to help you plan the following purposes that should be implemented to increase your protection significantly. These recommendations are based on evaluating many projects and environments and on the security experts’ knowledge. This enables you to use the available means and opportunities to increase your security.

b) Level 0

The first level mentioned is marked with the number zero. This is the state in which no action has yet been taken. The documentation that describes the entire build process must be available and up-to-date to reach this level. This enables an initial analysis of the environment to be secured. It can also be seen as a way of taking stock.

c) Level 1

The level with the number one requires first actions on the environment. The point here is to list all the components used to create a binary file. Not only are the dependencies used listed, but all other metadata is recorded if possible. There is the term SBOM, Software Bills Of Materials in the industry. However, the term can be broadened considerably here. Within JFrog Artifactory, there is “build info”, which means a superset of SBOM. All sorts of metadata are recorded here, such as the operating system and version of the build machine, environment variables and their contents, etc. I have created my video for this.

In this video, I’ll go over the details of SBOM and Build Info.

https://www.youtube.com/watch?v=ILgZexU07dc

Within the SLSA project, the term SBOM or Build Info is not explicitly mentioned. Instead, the requirement is described more generally, and it is also clearly pointed out that this level does not yet help against the compromise itself. It is only the basis for the follow-up and the comprehensive risk assessment. Based on this level, one can start with vulnerability management.

d) Level 2

Level two describes code versioning software in combination with a hosted build service. With this combination, you want to trust the build service and thus trust the origin of the generated files. Here you must take a critical look at where and who will offer these services. The fundamental consideration is based on the assumption that a specialized provider has a better grip on the issue of security within the build infrastructure than you can guarantee yourself. However, the manipulation of protection of the build service can be increased in various ways.

e) Level 3

With level three, auditors are now included in the considerations. Here, trained security specialists are used to check the existing build environment for possible vulnerabilities. As part of an accreditation process, the auditors determine whether specific standards are met to ensure the verifiability of the source and the integrity of the provenance (origin). This level offers better protection than all previous levels to ward off well-known threats. Preventing cross-build contamination is an example.

f) Level 4

In the SLSA project, level four is the highest currently achieved. To reach this level, the following conditions are set. The first requirement describes the necessity that two people constantly carry out all changes. This does not only refer to the changes made to the source code. The term is broader here and also includes all changes to the system and the configurations. This ensures that no intentional or unintentional compromise can occur at any point. Such a process has prevented most bugs, vulnerabilities and compromise attempts.

The second requirement relates to the technical environment. It is assumed here that it is a hermetic and reproducible creation process. However, a distinction should be made between the requirement for hermetic builds and reproducible builds. The former guarantees that all required dependencies are known and checked for vulnerabilities. Here, of course, it plays a role from which sources these dependencies were obtained. On the other hand, reproducible builds help detect changes and conduct subsequent analyses. However, the reproducibility of a build is not a mandatory part of the fourth level.

g) Limitations

Even if many supply chain threads are addressed with the SLSA project, there are some limitations that I would like to list here.

1) The list of all dependencies in a project can simply become too large to be able to fully record and evaluate them. That can depend on different things. The point that shows this most clearly is based on the possibility of some dependency management systems not only specifying explicit dependency versions. In some scenarios, you can also define a version range. This now results in an exponential increase in all possible version combinations.

2) The number of all components to be backed up can exceed the available capacities. The teams then have to decide which elements are subjected to safety analysis.

3) The SLSA security levels are not transitive. This means that this component itself consists of components that have a lower security level. So here, you have to pay very close attention to the details.

4) Even if automation is the solution in many areas, there may be areas where this will not be possible. Again, this refers in many areas to closed-source projects. Here, some analyzes are simply not feasible. It must then be decided how this dependency will be classified on a case-by-case basis.

**What are the requirements? **

All the requirements mentioned here are a little more detailed in reality. An overview of the requirements was provided to guide what exactly is meant by each point. Here, you can see which condition belongs to which level and how this is understood. Then the details are discussed, such as what difference it makes if you only use a building service or if you also use the “Build as Code” strategy. All the details are too extensive here, so I refer to the section on the project’s website.

Supply Chain Threads

**a) Overview **

To better understand the project and its limitations, you should look at the individual types of attacks on a typical software development environment. A distinction is generally made between source, image and dependency threads. The SLSA project also delimits precisely what it includes and what it does not.

The following overview comes directly from the project and will be further developed if necessary. At this point, one must not forget that this project is only in the alpha phase (early 2022). However, these types of attacks mentioned are timeless and apply to almost every development environment.

Let’s get to the source threads. This type of attack aims to compromise the source code itself.

b.1) Bypass Code Review

The attack called “Bypass Code Review” describes a scenario in which an attempt is made to bypass the control mechanisms and directly add compromised source code to the project. Here public paths are taken to reach the destination. Such attacks were carried out on the Linux kernel, for example, via the mailing list. The best and probably simplest way to thwart such attacks is to have all source code changes checked independently by at least two people. However, at this point, one must also find ways to prevent mutual “covering up”.

b.2) Compromised Source Control System

If importing compromised source code does not work, you can try to attack the CVS itself. Attempts are being made here to make changes to the source code on the server, bypassing the official channels. A prominent example is a successful attack on the source code of PHP. The self-managed system was attacked, and the changes were imported directly as a commit. Only securing the system helps against this type of attack. To manage such a system yourself, you must have the necessary resources and knowledge. In this case, it is often easier and cheaper to use a managed system with Github.

Next, we come to the build threads.

b.3) Modified Source Code after Source Control

The source code cannot only be changed where it is stored. When a Build process is started, the source code is fetched from the specified source. The source text on the build page can also be modified before the translation process starts. One attack in which this approach was used is the “Webmin” project. Here, additional and modified source texts were injected into the translation process. The agents responsible for carrying out the individual build steps must be secured to prevent this attack. Here, too, one has to weigh up how much of this work is carried out by oneself since how safe such an environment depends primarily on one’s own abilities.

b.4) Compromised Build Platform

Since the built environment is part of the value chain, this part must also be secured classically. Here, too, it is again the case that one’s own abilities in ​​safety must be assessed. A well-known representative of this type of attack is the Solarwinds Hack. Here, the build route was modified to compromise the newly created binary with each run.

b.5) ​​Using Bad Dependencies - Dependency Thread

At this point, I would like to insert the dependency threads briefly. Dependencies are specifically modified and then offered. The security of the repositories plays a central role here. I’ll come back to that in a moment. However, this attack pattern refers to external locations where dependencies are fetched to process them. Particular attention should be paid to assessing the integrity of caches and mirrors of known official bodies. Often the “official” mirror servers of publicly accessible repositories do not have the same financial resources as the original. This causes attacks to be placed there.

You may be offered modified versions of known commonly used dependencies when you dock at such a place.

b.6) Bypassed CI/CD

Let’s get back to the build threads. Sometimes it is easier to bypass the CI/CD environment at an appropriate point. Here, the topics within a development path are selected where a transfer between the build server and the associated repositories is not sufficiently secured. The attack involves offering a recipient such as a repository a modified binary to make it look like it is from the CI environment. Malicious code is injected parallel to its own build infrastructure.

b.7) Compromised package repo

A component repository is also part of the infrastructure and can fall victim to an attack. As with all other elements, protection is necessary here. Attackers try to gain access to the repositories to replace the binaries of known and frequently used dependencies with modified versions. Depending on your ability to harden such a system against attacks, you have to decide whether an externally managed alternative will increase the security standard of your value chain.

b.8) Using a bad package

We come to the last attack variant mentioned here. In this scenario, known dependencies are taken as a template and then their own modified versions are offered in a public repository. Here, an attempt is made to give the own variant a name close to the original name and depicts typical spelling errors. This dependency can still be resolved if a misspelling of the original name is used in a version definition. Only in this case a modified version of it will be loaded.

The only way to protect yourself against this attack is to carefully check how each individual dependency has been defined in your own projects.

Project Persia Overview

We have seen which general types of attacks on a classic software development environment are possible. These are always generic attack patterns that can be used independently of the current products. Others also saw this as an idea to develop a general strategy to counter this. The language here is from the open-source project Persia, which is currently (early 2022) in the alpha phase. We are close to officially accepting it as an incubator in the CD Foundation. The project website ishttps://pyrsia.io.

The basic idea of ​​this project is that there will be a network of trust for open-source projects. A P2P network consisting of build and distribution nodes should exist where the sources are fetched, built and offered decentrally. In addition to the classic advantages such as reliability and better utilisation of the network resources of a P2P network, increased security should also be achieved with the binaries themselves. The types of attacks on source code management software are excluded. But all subsequent attack patterns can be repelled in this way.

Unfortunately, it is not possible to fully describe the project. But at this point, I would like to draw attention to my video about this project. (https://pyrsia.io/)

https://youtu.be/M3QQww1JzlU

Cheers Sven

]]>DevSecOpsSecurityUncategorizedhttps://svenruppert.com/posts/the-power-of-jfrog-build-info-build-metadata/Fri, 08 Oct 2021 13:42:05 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/the-power-of-jfrog-build-info-build-metadata/Intro This article will take a detailed look at what the term build-info is all about and why it will help us protect against attacks such as the Solarwinds Hack.<![CDATA[

Intro

This article will take a detailed look at what the term build-info is all about and why it will help us protect against attacks such as the Solarwinds Hack.

What is the concept behind the term - build-info?

Let’s start at the very beginning and clarify the basic principle behind the term build-info. The term build-info has been coined for many years by the company JFrog, among others. This is a particular type of repository.

This repository stores the information that describes the context that led to the creation of a binary file. With this information, you can now achieve a wide variety of things.

https://youtu.be/ILgZexU07dc

What components does build-info consist of?

The content of a build-info is not strictly defined. Instead, the approach that applies is that the more, the better. Of course, you have to proceed with caution here too. All possible parameters are collected. In addition to the date and time, the system on which the process was run, which operating system was used in which patch level, to active environment variables, compiler switches and library versions.

The challenge is actually that it is not known which information will later be helpful and expedient. For this reason, more rather than less should be saved.

Why do we actually need a build-info?

The task of a build-info is to enable the observation, or rather, the analysis of a past situation. There can be a variety of reasons for this. For example, it can be used to improve quality, or it can be the basis for reconstructing a cyber attack that has taken place. And with that, we come straight to the event that got everything rolling in the recent past.

Trigger - SolarWinds Hack

One of the others will have heard or read something about it. We are talking about one of the most significant cyberattacks that have ever taken place. It’s the SolarWinds Hack. Here it was not the final target that was attacked directly, but a point in the supply chain. SolarWinds is a software company that provides a product for managing network infrastructure. With just over 300,000 customers worldwide, this software’s automatic update process has been the target of the attack. It was not the update process itself that was compromised, but the creation of the binaries that will be distributed with this update process. The attack took place on the company’s CI route to immediately infect the newly created binaries with each build. Here the CI route was manipulated so that another component was added to the binary to be generated. This component can be thought of as a kind of initial charge. As soon as this has been ignited or activated, further components are dynamically reloaded. As a result, each infection had different forms. These files were then offered to all customers by means of an automatic update. Thus, over 15,000 systems were infiltrated within a short time.

Reaction - Executive Order of Cybersecurity

Since there were many well-known US companies, US organizations and US government institutions among the victims, the question arose of how to counter such a threat from the US state in the future. The US government has decided that one begins with the complete cataloguing of all software components in use, including all their constituent parts. This obligation to provide evidence was formulated in the “Executive Order of Cybersecurity”. However, when I first heard about an executive order, I wasn’t sure what that actually meant.

What is an executive order?

An executive order is a decree of the US President that regulates or changes internal affairs within the state apparatus. You can think of it as a US president like a managing director of a company who can influence his company’s internal processes and procedures. In doing so, no applicable law can be circumvented or changed. With such an executive order, no law can be changed, stopped or restricted. But it can change the internal processes of the US authorities very drastically. And that’s exactly what happened with this Executive Order of Cybersecurity, which has directly impacted the US economy. Every company that works directly or indirectly for the state must meet these requirements to continue doing business with the US authorities.

What is the key message of the Executive Order?

The Executive Order of Cybersecurity contains a little more text, the content of which I would like to shorten here and reproduce without guaranteeing legal correctness.

This arrangement aims to record the software operated by or for the US authorities in its entirety. This means that all components of the software used must be documented. This can be thought of as follows; If you want to bake a cake, you need a list of ingredients. However, listing these would not meet the requirements, as knowledge of all elements down to the last link is required. In our example, an ingredient that consists of more than one component would have to provide a list of all existing parts. This is how you get all the components used that are needed for the cake. If we now transfer this to the software, it is necessary to record all direct and indirect dependencies. The list of all components is then called - SBOM (Software Bills Of Material)

And what does that mean for me?

Now I am not directly working for an organ of the US government. But it will still reach me sooner or later. Because through the indirection, all possible economic sectors worldwide will be affected. It can also affect me in the private sector. If I have an open-source project that is used indirectly in this environment, it can only continue if I prepare the project accordingly. Long story short; It will come our way in whatever way it will happen.

What are the general requirements for a build-info?

Let’s get back to the build-info. A build-info is a superset of an SBOM (Software Bills Of Material). So the dependencies are catalogued, and additional information from the runtime environment in which the binary is generated is recorded. However, there are also some requirements for such build-info. By this, I mean properties that must be present in order to be able to deal with this information in a meaningful way.

Accessibility:

The information that is collected must be quickly and easily accessible and therefore usable. A central storage location such as Artifactory is suitable here, as all binaries and dependencies are also stored here. You also have all the meta-information of a respective binary file at this location.

Immutability:

When information is collected, it makes sense to store it in such a way that it can no longer be changed. This promotes the acceptance of the data situation and gives particular security when evaluating a case.

Combinability:

The static data are also suitable for enriching them with current secondary data. In this case, I am talking about the vulnerabilities that can be found in the dependencies used. In contrast to the static information, this image has to be constantly updated. In practical use, this means that the data must be stored in a machine-readable manner in order to enable the connection of other data sources. In the case of Artifactory, it is the data from JFrog Xray that is displayed here.

How can a build-info be generated?

It would be best if you had the following things.

First, the JFrog CLI generates the build-info and second, it is possible to save this information permanently. You can do this very quickly with the Build-Info Repositories from Artifactory. But one step at a time.

The process for generating a build-info is as follows.

In the following example, we assume that we are going to build a Java project with maven. Here you have to get involved in the build process. You can do that quite easily with the JFrog CLI tools. These represent a wrapper for the build process used in each case. In our example, it is maven. To install the JFrog CLI Tools, you can go on the websitehttps://jfrog.com/getcli/ choose the installer that is suitable for it. After installing and configuring the JFrog CLI Tools, you can start the build via the CLI on the command line. After a successful build, you can find the build-info as a JSON file in the target directory. This file can be viewed with a text editor to get a feel for the wealth of information that is stored here. This file is regenerated with each build. This means that there is a 1: 1 relationship between the build cycle and the binaries generated in the process. Using the CLI, you can then transfer this information to the Artifactory build repository and evaluate it there using a graphical user interface.

How can I link vulnerability scans to the build-info?

If you are now within Artifactory (https://bit.ly/SvenYouTube ), look at the build-information; you also get the current information regarding the known vulnerabilities. This is an example of how the build-information can be enriched with further system information.

Since there is not enough space at this point to clarify the generation and evaluation, I refer to a demo project in which I have stored the steps in the README, including detailed explanations of the individual practical steps.

If you want, you can also register for one of my free DevSecOps workshops. You are also welcome to speak to me directly if you’re going to do this in a free community or company event.

The URL for the demo project is:

https://github.com/Java-Workshops/JFrog-FreeTier-JVM

Have fun trying it out.

Cheers, Sven!

]]>DevSecOpsSecurityhttps://svenruppert.com/posts/solarwinds-hack-and-the-executive-order-from-mr-biden-and-now/Tue, 27 Jul 2021 11:10:15 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/solarwinds-hack-and-the-executive-order-from-mr-biden-and-now/ In the past two years, we have had to learn a lot about cybersecurity. The new attack vectors are becoming more and more sophisticated and are directed more and more against the value chain in general. But what does that mean for us? What can be done about it, and what reactions have the state already taken?<![CDATA[

In the past two years, we have had to learn a lot about cybersecurity. The new attack vectors are becoming more and more sophisticated and are directed more and more against the value chain in general. But what does that mean for us? What can be done about it, and what reactions have the state already taken?

Let’s start with the story that got all of this rolling and made sure that the general attention was drawn to the vulnerabilities of the available IT infrastructure. We’re talking about the SolarWinds Hack. What happened here, and what is more critical; What are the lessons learned from this incident?

https://youtu.be/ClYhATBlBl0

It is essential to know that the SolarWinds company produces software that is used to manage network infrastructure. With the name “Orion Platform, " this software should help manage and administer the network components efficiently. If you look at the product description on the SolarWinds website, you can read that the software can monitor, analyse, and manage the network infrastructure. And that’s exactly what the customers did. The company itself has around 300,000 customers worldwide, including individual companies and government organisations and corporations from a wide variety of areas. To always stay up to date, the software platform includes an automatic update mechanism. And that was exactly what the attackers were after. They saw in the SolarWinds company a multiplier for their own activities. But how can you go about this? The attackers obtained the necessary software tools by breaking into the FireEye company and infiltrating the SolarWinds network. The goal was the software development department in which the binaries of the Orion platform are created. Here, the CI routes have been manipulated to include compromised binaries in every build of the software. As a result, the company produced these binaries and put them into circulation through the automatic update process. In this way, around 18,000 targets could be infiltrated and compromised within a short time.

What does that mean for us now?

There are two angles here. The first is from the consumer’s point of view. However, for your own production, it must be ensured that no compromised binaries are used. That means that all, and here I mean all binaries used, must be checked. These are the dependencies of the application and the tools used in the production process. This includes the tools such as the CI server used, e.g. Jenkins, or the operating system components of the infrastructure and Docker images.

The second perspective represents the point of view that one has as a software distributor. This means anyone who distributes binaries in any form. This includes customers in the classic sense and other departments that may be considered consumers of the binaries created. The associated loss of trust can cause a company severe financial damage.

What can we do to protect ourselves?

If you look at the different approaches to arm yourself against these threats, you can see two fundamental differences.

DAST - Dynamic Application Security Testing

An application can be subjected to a black box test. This is called DAST, Dynamic Application Security Testing. Here you go the way a hacker would do it. The application is attacked via the available interaction points. Attack vectors based on the most common vulnerabilities are usually used here. SQL injection can be cited as an example. These attack patterns are initially independent of the technology used and represent typical security holes in web applications.

This approach has advantages as well as disadvantages. A serious disadvantage is that the tests cannot be started until the application has been developed. This means that the measures to eliminate weak points found are more costly than is the case with other approaches. On the other hand, a significant advantage of this approach is that the dynamic context can also be checked when testing the production system. Examples include weak configurations, open ports, and more.

SAST - Static Application Security Testing

The counterpart to the dynamic tests is the static tests. These tests analyse the individual components of the infrastructure involved. This includes not only the applications created but also the components involved in operating and creating the application.

A disadvantage of the SAST method is that it can only search for known security vulnerabilities. However, this point is put into perspective in that the number of known security gaps increases steadily. Still, it is also assumed that the number of known security gaps exceeds the number of unknown security gaps. Another advantage of the SAST method is that it checks the licenses used. This area also contributes to the security of the company, albeit in a legal sense.

So what has the most significant effect when you start with the topic of security?

When looking at the different approaches, it becomes clear that the proportion of directly or indirectly used binaries makes up the most significant amount in a technology stack. This means that you should also start securing precisely these parts. Unfortunately, there is only one way to ensure with SAST that it is really 100% checked directly. All other methods do not achieve this coverage, or if they do, then only in sporadic exceptional cases.

However, it should be noted that one understands the interdependencies. This is the only way to identify and extract the possible attack vector. Here it is of essential importance that not only the binaries themselves are examined, but also the associated meta-information is evaluated. Within software development, there are two points for these analyses where such an analysis fits very well. The first point that most people will think of is the CI segment. Here the CI server can carry out the analysis during the execution of a build pipeline and all the checks and also take over the reporting. If the violations are too massive, the build is cancelled. The second position, which is ideally suited for such an analysis, is directly in the IDE. Here it can already be determined during the definition whether the license used is permitted. The known vulnerabilities can also be extracted and thus provide enough information to decide whether this version of this dependency may be used.

Lesson Learned - what we learned from the SolarWinds Hack.

We learned the following things from SolarWinds. First, the attacks are no longer aimed at the final targets themselves but against the suppliers. Here one tries to reach multipliers who massively increase the effectiveness of this attack. Likewise, you no longer only have to protect yourself against your use of compromised binaries. The newly added component means that the self-generated binaries must also be secured. Nobody wants to become a distributor of infected software. It follows that all parts involved must be subjected to permanent checks.

Reactions - The Executive Order from Mr Biden

One or the other has undoubtedly noticed that the US President gave this Executive Order of Cybersecurity. But what exactly does that mean, and who can even do something like that under what conditions?

In the United States, the president - the “executive branch” of the US government - can issue what is known as an executive order. Much like a CEO in a company, the executive branch has the power to make some unilateral decisions that affect the behaviour of the US government. These orders must be limited to government conduct and policies and not to general laws or statutes that encroach on the rights of any individual or state. If the executive transgresses in this area, the US Supreme Court or other legal remedies can appeal and revoke all measures. Only the president - due to his status as head of state - can issue an executive order. No other officer is allowed to do this.

Significantly, executive orders are mostly limited to how the US government operates internally. For example, an executive order might not simply resolve to raise taxes in a state or issue far-reaching commercial guidelines. However, executive orders are legally binding on the behaviour of the US government if they do not violate fundamental rights or make something unconstitutional. Because of this, these commands can be found to change the policy from the president to the president as, like a CEO, they set some of the government’s agendas. In this case, the Executive Ordinance refers to how the US government monitors and regulates its software usage and sets out requirements for how that software must be documented to be sold to the government. It doesn’t dictate trading on a broad scale.

Since the US government can define its way of working, it was decided that the SBOM requirements (in preparation) will be the only way to approve software purchases - to get an idea of ​​the volume, you have to know it is one amount in the tens of billions per year. This means that any company that wants to sell to US government agencies, likely state and local agencies, and anyone who works with resellers who sell to the government and more, must meet all of these new standards.

When the European Union agreed on comprehensive data protection and data security laws with the GDPR, the effect was felt worldwide and quickly. In a globally networked economy, decisions by the world’s largest GDP producers can set standards for other areas. With the GDPR, companies in the US and elsewhere had to immediately revise and update software and processes to continue doing business in the EU. It became a worldwide rule so as not to lose EU treaties. Likewise, the US cybersecurity order is believed to be reversing, with U.S.-based global corporations influencing global software security behaviour.

Wow - what now?

What does that mean for me as a software developer? Well, the effects at this point are not as severe as you might want to imagine. In the end, it is said that a complete list of all contained dependencies must be created. This means the direct and indirect dependencies. To obtain such a dependency graph, one can, for example, use the tools that already do this. It means the tools that can generate a complete dependency graph, for example. For this, you need a logical point through which all dependencies are fetched. A tool such as Artifactory is best, as this is where the knowledge of all dependency managers used comes together. Devices such as the vulnerability scanner JFrog Xray can then access this information, scan for known vulnerabilities and licenses used, and extract the entire dependency graph. This information can thus be generated as build information by the CI route (e.g. JFrog Pipelines) and stored in a build repository within Artifactory.

Conclusion

We have seen that the new qualities of attacks on software systems, in general, will lead to advanced countermeasures. What is meant here is a complete analysis of all artefacts involved and produced. The tools required for this must combine the information into aggregated impact graphs across technology boundaries. One point within software development that is ideal for this is the logical point via which all binaries are fed into the system. However, the components alone are insufficient since all indirect elements can only be reached and checked by understanding the associated meta-information.

The tools used for this can then also be used to create the SBOMs.

In my opinion, it is only a matter of time before we also have to meet this requirement if we do not want to suffer economic disadvantages from the Executive Order of Cybersecurity.

If you want to find out more about the topic or test the tools briefly mentioned here, don’t hesitate to contact me.

In addition to further information, we at JFrog offer free workshops to prepare for these challenges in a purely practical manner.

And if you want to know more immediately, you are welcome to go to my YouTube channel. There I have other videos on this topic.

It would be nice to leave a subscription on my YouTube channel as a small thank you.https://www.youtube.com/channel/UCNkQKejDX-pQM9-lKZRpEwA

Cheers Sven

]]>DevSecOpsSecurityhttps://svenruppert.com/posts/what-is-the-difference-between-sast-dast-iast-and-rasp/Mon, 19 Jul 2021 15:34:30 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/what-is-the-difference-between-sast-dast-iast-and-rasp/ Intro: In this post, we’re going to look at the differences between the various cybersecurity defence techniques. Here you can identify four main groups, which we will go through briefly one after another to illustrate the advantages and disadvantages.<![CDATA[

Intro:

In this post, we’re going to look at the differences between the various cybersecurity defence techniques. Here you can identify four main groups, which we will go through briefly one after another to illustrate the advantages and disadvantages.

https://youtu.be/sW7mTNVIUhE

SAST - Static Application Security Testing

SAST describes the process in which the components of an application are subjected to a static analysis. This approach not only searches for security gaps but also determines the licenses for the individual elements. In the following, however, I will only deal with the consideration of vulnerabilities in this post.

The term static comes from the fact that only the static context is used for this evaluation. All dynamic or context-related assessments are not possible with this method. SAST can be used to analyze the entire tech stack of an application if access to these components is possible, which is also one of the advantages of SAST compared to other analysis methods. The only exception is the IAST procedure, which we will come back to later.

What exactly happens now with the SAST procedure? SAST is available in two forms. The first that I would like to mention relates to checking your source code. Here the attempt is made to determine whether there are patterns that make a later vulnerability easier or even possible in the first place. For example, it searches for calls to critical libraries or identifies unsafe handling of resources. However, one honestly has to admit at this point that, first of all, your source code will be by far the smallest part of the overall application in the vast majority of projects. Second, the analysis methods in this area are still at a very early stage of development.

When using SAST, it makes a lot more sense to deal with all the other components first. Therefore, all binary files of the entire application environment are meant here. This also includes all elements that play a role in the development, such as the CI server and the other tools used.

DAST - Dynamic Application Security Testing

The DAST analysis method does precisely the opposite in comparison to the SAST method. Here the application is viewed as a black box. As soon as an executable version of the application is available, it is attacked through automatically executed cyber attacks. In most cases, the general attack patterns are based on the Most Common Vulnerabilities defined by the OSWAP. Defining your attack vectors is either not provided for in the tools I know or requires very detailed knowledge in this area. For developers who do not use these tools daily and intensively, their use is limited to the predefined attack vectors. It is a good thing with this procedure that unknown security gaps can also be identified, as long as they are based on the Most Common Vulnerabilities. Some manufacturers extend this approach with AI or ML techniques. This approach using AI and ML techniques is still in a very early stage of development, and the currently available potential cannot be foreseen.

Unfortunately, the method can only be used sensibly against the production environment, as this is the only way to identify all gaps based on configuration deficiencies. Therefore, in contrast to the SAST process, it makes no sense to use it within the CI route.

IAST - Interactive Application Security Testing

IAST uses software tools to evaluate application performance and identify vulnerabilities. IAST takes an “agent-like” approach; H. Agents and sensors run to continuously analyze application functions during automated tests, manual tests, or a mixture of both.

The process and feedback occur in real-time in the IDE, Continuous Integration (CI) environment or quality assurance or during production. The sensors have access to:

• All source code
• Data and control flow
• System configuration data
• Web components
• Backend connection data

The main difference between IAST, SAST, and DAST is that it runs inside the application.

Access to all static components as well as the runtime information enables a very comprehensive picture.

It is a combination of static and dynamic analysis. However, the part of the dynamic analysis is not a pure black-box test as it is implemented at DAST.

Some reasons for using IAST:

Potential problems are identified earlier, so IAST minimizes the cost of eliminating potential costs and delays.

This is due to a “shift left” approach, meaning it is carried out in the early stages of the project life cycle.

Similar to SAST, the IAST analysis provides complete lines of data-rich code so that security teams can immediately lookout for a specific bug.

With the wealth of information that the tool has access to, the source of vulnerabilities can be precisely identified.

Unlike other dynamic software tests, IAST can be easily integrated into Continuous Integration and Deployment (CI / CD) pipelines.

The evaluations take place in real-time in the production environment.

On the other hand:

IAST tools can slow down the operation of the application.

This is based on the fact that the agents change the bytecode themselves. This leads to a lower performance of the overall system.

The change itself can, of course, also lead to problems in the production environment.

The use of agents naturally also represents a potential source of danger since these agents can also be compromised.

- See Solarwinds Hack

RASP - Runtime Application Self Protection

RASP is the approach to secure the application from within.

The backup takes place at runtime and generally consists of looking for suspicious commands when they are executed.

What does that mean now?

With the RASP approach, you can examine the entire application context on the production machine and real-time.

Here all commands that are processed are examined for possible attack patterns.

Therefore, this procedure aims to identify existing security gaps and attack patterns and those that are not yet known.

Here it goes very clearly into the use of AI and ML techniques.

RASP tools can usually be used in two operating modes.

The first operating mode (monitoring) is limited to observing and reporting possible attacks.

The second operating mode (protection) then includes implementing defensive measures in real-time and directly on the production environment.

RASP aims to fill the gap left by application security testing and network perimeter controls.

The two approaches (SAST and DAST) do not have sufficient visibility into real-time data and event flows to prevent vulnerabilities from sliding through the verification process or block new threats that were overlooked during development.

RASP is similar to Interactive Application Security Testing (IAST); the main difference is that IAST focuses on identifying vulnerabilities in the applications, and RASPs focus on protecting against cybersecurity attacks that can exploit these vulnerabilities or other attack vectors.

The RASP technology has the following advantages:

First, RASP complements SAST and DAST with an additional layer of protection after the application is started (usually in production).

It can be easily applied with faster development cycles.

Unexpected entries are checked and checked.

It enables you to react quickly to an attack by providing comprehensive analysis and information about the possible vulnerabilities.

However, RASP tools have certain disadvantages:

By sitting on the application server, RASP tools can adversely affect application performance.

In addition, the technology (RASP) may not be compliant with regulations or internal guidelines,

which allows the installation of other software or

the automatic blocking of services is permitted.

The use of this technology can also lead to the people involved believing themselves to be false security.

The application must also be switched offline until the vulnerability is eliminated.

RASP is not a substitute for application security testing because it cannot provide comprehensive protection.

While RASP and IAST have similar methods and uses, RASP does not perform extensive scans but instead runs as part of the application to examine traffic and activity. Both report attacks as soon as they occur; with IAST, this happens at the time of the test, while with RASP, it takes place at runtime in production.

Conclusion

All approaches result in a wide range of options for arming yourself against known and unknown security gaps. Here it is essential to reconcile your own needs and those of the company for the future direction.

Conclusion RASP:

We have seen that with RASP, there is an approach in which the application can protect itself against attacks at runtime. The permanent monitoring of your activities and the data transferred to the application enable an analysis based on the runtime environment. Here you can choose between pure monitoring or alerting and active self-protection. However, software components are added to the runtime environment with RASP approaches to manipulate the system independently. This has an impact on performance. With this approach, RASP concentrates on the detection and defence of current cyber attacks. So it analyzes the data and user behaviour in order to identify suspicious activities.

Conclusion IAST:

The IAST approach combines the SAST and DAST approach and is already used within the SDLC, i.e. within the development itself. This means that the IAST tools are already further “to the left” compared to the RASP tools. Another difference to the RASP tools is that IAST consists of static, dynamic and manual tests. Here it also becomes clear that IAST is more in the development phase. The combination of dynamic, static and manual tests promises a comprehensive security solution. However, one should not underestimate the complexity of the manual and dynamic security tests at this point.

Conclusion DAST:

The DAST approach focuses on how a hacker would approach the system. The overall system is understood like a black box, and the attacks occur without knowing the technologies used. The point here is to harden the production system against the Most Common Vulnerabilities. However, one must not forget at this point that this technology can only be used at the end of the production cycle.

Conclusion SAST:

If you have access to all system components, the SAST approach can be used very effectively against known security gaps and license problems. This procedure is the only guarantee that the entire tech stack can be subjected to direct control. The focus of the SAST approach is on static semantics and, in turn, is completely blind to security holes in the dynamic context. A huge advantage is that this approach can be used with the first line of source code.

My personal opinion:

My personal opinion on these different approaches is as follows:

If you start with DevSecOps or security in IT in general, the SAST approach makes the most sense. This is where the greatest potential threat can be eliminated with minimal effort. It is also a process that can be used in all steps of the production line. Only when all components in the system are secured against known security gaps do the following methods show their highest potential. After introducing SAST, I would raise the IAST approach and, finally, the RASP approach. This also ensures that the respective teams can grow with the task and that there are no obstacles or delays in production.

Cheers Sven

]]>DevSecOpsSecurityhttps://svenruppert.com/posts/the-lifeline-of-a-vulnerability/Fri, 25 Jun 2021 16:17:29 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/the-lifeline-of-a-vulnerability/ Intro Again and again, we read something in the IT news about security gaps that have been found. The more severe the classification of this loophole, the more attention this information will get in the general press. Most of the time, you don’t even hear or read anything about all the security holes found that are not as well known as the SolarWinds Hack, for example. But what is the typical lifeline of such a security gap?<![CDATA[

Intro

Again and again, we read something in the IT news about security gaps that have been found. The more severe the classification of this loophole, the more attention this information will get in the general press. Most of the time, you don’t even hear or read anything about all the security holes found that are not as well known as the SolarWinds Hack, for example. But what is the typical lifeline of such a security gap?

https://youtu.be/2_b0KxUl4vs

Vulnerability was generated until found

Let’s start with the birth of a vulnerability. This birth can be done in two differently motivated ways. On the one hand, it can happen to any developer that he creates a security hole by an unfortunate combination of source code pieces. On the other hand, it can also be based on targeted manipulation. However, this has essentially no effect on the further course of the lifeline of a security vulnerability. In the following, we assume that a security hole has been created and that it is now active in some software. These can be executable programs or libraries offered that are integrated into other software projects as a dependency.

Found until Public available

In most cases, it is not possible to understand precisely when a security hole was created. But let’s assume that there is a security hole and that it will be found. It clearly depends on which person or which team finds this weak point. This has a severe impact on the subsequent course of history. Let’s start with the fact that this vulnerability has been found by people who are interested in using this vulnerability themselves or having it exploited by other parties. Here the information is either kept under lock and key or offered for purchase at relevant places on the Internet. There are primarily financial or political motives here, which I do not want to go into here. However, at this point, it can be clearly seen that the information is passed on at this point in channels that are not generally available to the public.

However, if the security gap is found by people or groups who are interested in making the knowledge about it available to the general public, various mechanisms now take effect. However, one must not forget that commercial interests will also play a role in most cases. However, the motivation is different. If the company or the project itself is affected by this vulnerability, there is usually an interest in presenting the information relatively harmless. The feared damage can even lead to the security gap being fixed, but knowledge about it further is hidden. This approach is to be viewed as critical, as it must be assumed that there will also be other groups or people who will gain this knowledge.

But let’s assume that people who are not directly involved in the affected components found the information about the vulnerability. In most cases, this is the motivation to sell knowledge of the vulnerability. In addition to the affected projects or products, there are also providers of the vulnerability databases. These companies have a direct and obvious interest in acquiring this knowledge. But to which company will the finder of the vulnerability sell this knowledge? Here it can be assumed that there is a very high probability that it will be the company that pays the better price. This has another side effect that affects the classification of the vulnerability. Many vulnerabilities are given an assessment using the CVSS. Here, the base value is made by different people. Different people will have other personal interests here, which will then also be reflected in this value.

Here I refer to the blog post about CVSS - explained.https://svenruppert.com/2021/04/07/cvss-explained-the-basics/

Regardless of the detours via which the knowledge comes to the vulnerabilities databases. Only when the information has reached one of these points can one assume that this knowledge will be available to the general public over time.

Public available until consumable

However, one fact can be seen very clearly at this point. Regardless of which provider of vulnerabilities you choose, there will only ever be a subset of all known vulnerabilities in this data set. As an end consumer, there is only one sensible way to go here. Instead of contacting the providers directly, you should rely on integrators. This refers to services that integrate various sources themselves and then offer them processed and merged. It is also essential that the findings are processed so that further processing by machines is possible. This means that the meta-information such as the CVE or the CVSS value is supplied. This is the only way other programs can work with this information. The CVSS value is given as an example. This is used, for example, in CI environments to interrupt further processing when a particular threshold value is reached. Only when the information is prepared in this way and is available to the end-user can this information be consumable. Since the information generally represents a considerable financial value, it can be assumed in the vast majority of cases that the commercial providers of such data sets will have access to updated information more quickly than freely available data collections.

Consumable until running in Production

If the information can now be consumed, i.e. processed with the tools used in software development, the storyline begins in your projects. Whichever provider you have decided on, this information is available from a particular point in time, and you can now react to it yourself. The requirement is that the necessary changes are activated in production as quickly as possible. This is the only way to avoid the potential damage that can result from this security vulnerability. This results in various requirements for your software development processes. The most obvious need is the throughput times. Only those who have implemented a high degree of automation can enable short response times in the delivery processes. It is also an advantage if the team concerned can make the necessary decisions themselves and quickly. Lengthy approval processes are annoying at this point and can also cause extensive damage to the company.

Another point that can release high potential is the provision of safety-critical information in all production stages involved. The earlier the data is taken into account in production, the lower the cost of removing security gaps. We’ll come back to that in more detail when the shift-left topic is discussed.

Another question that arises is that of the effective mechanisms against vulnerabilities.

TestCoverage is your safety-belt; try Mutation Testing

The best knowledge of security gaps is of no use if this knowledge cannot be put to use. But what tools do you have in software development to take efficient action against known security gaps? Here I would like to highlight one metric in particular: the test coverage of your own source code parts. If you have strong test coverage, you can make changes to the system and rely on the test suite. If a smooth test of all affected system components has taken place, nothing stands in the way of making the software available from a technical point of view.

But let’s take a step back and take a closer look at the situation. In most cases, changing the version of the same dependency in use will remove known vulnerabilities. This means that efficient version management gives you the agility you need to be able to react quickly. In very few cases, the affected components have to be replaced by semantic equivalents from other manufacturers. And to classify the new composition of versions of the same ingredients as valid, strong test coverage is required. Manual tests would go far beyond the time frame and cannot be carried out with the same quality in every run. But what is strong test coverage?

I use the technique of mutation testing. This gives you more concrete test coverage than is usually the case with the conventional line or branch coverage. Unfortunately, a complete description of this procedure is not possible at this point.

However, if you want to learn more about mutation testing, visit the following URL. This video explains the theoretical and practical part of the mutation test for Java and Kotlin.

https://www.youtube.com/watch?v=6Vej7YEOF8g

The need for a single point that understands all repo-types

If we now assume that we want to search for known security vulnerabilities in the development and the operation of the software, we need a place where we can carry out the search processes. Different areas are suitable here. However, there is a requirement that enables an efficient scan across all technologies used. We are talking about a logical central point through which all binaries must go. I don’t just mean the jar files declared as dependencies, but also all other files such as the Debian packages or Docker images. Artifactory is suitable for this as a central hub as it supports pretty much all package managers at one point. Because you have knowledge of the individual files and have the metadata, the following things can also be evaluated.

First of all, it is not only possible to capture the direct dependencies. Knowing the structures of the package managers used means that all indirect dependencies are also known. Second, it is possible to receive cross-technology evaluations. This means the full impact graph to capture the practical meaning of the individual vulnerabilities. The tool from JFrog that can give an overview here is JFrog Xray, and we are directly connected to JFrog Artifactory. Whichever tool you choose, it is crucial that you don’t just scan one technology layer. Only with a comprehensive look at the entire tech stack can one guarantee that there are no known security gaps, either directly or indirectly, in production.

Conclusion

Now we come to a conclusion. We have seen that we have little influence on most sections of a typical lifeline of an IT security vulnerability. Actually, there are only two sections that we can influence directly. On the one hand, it is the quickest and most comprehensive possible access to reliable security databases. Here it is essential that you not only entrust yourself to one provider but rather concentrate on so-called “mergers” or “aggregators”. The use of such supersets can compensate for the economically motivated vagueness of the individual providers. I named JFrog Xray as an example of such an aggregator. The second section of a typical lifeline is in your own home. That means, as soon as you know about a security gap, you have to act yourself—robust automation and a well-coordinated DevSecOps team help here. We will deal with precisely this section from the “security” perspective in another blogpost. Here, however, we had already seen that strong test coverage is one of the critical elements in the fight against vulnerabilities. Here I would like to refer again to the test method “Mutation Testing”, which is a very effective tool in TDD.

And what can I do right now?

Aou can, of course, take a look at my YouTube channel and find out a little more about the topic there. I would be delighted to welcome you as my new subscriber. Thanks!

https://www.youtube.com/@OutdoorNerd

Happy Coding

Sven

]]>DevSecOpsSecurityhttps://svenruppert.com/posts/cvss-explained-the-basics/Wed, 07 Apr 2021 12:20:21 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/cvss-explained-the-basics/ Intro What is the Common Vulnerability Scoring System short called CVSS, who is behind it, what are we doing with it and what a CVSS Value means for you? I will explain how a CVSS Score is calculated, what the different elements of it mean and what are the differences between the different CVSS versions.<![CDATA[

Intro

What is the Common Vulnerability Scoring System short called CVSS, who is behind it, what are we doing with it and what a CVSS Value means for you? I will explain how a CVSS Score is calculated, what the different elements of it mean and what are the differences between the different CVSS versions.

The Basic Idea Of CVSS

The basic idea behind CVSS is to provide a general classification of the severity of a security vulnerability. This is about the classification and evaluation of weak points. But what does the abbreviation CVSS mean?

What does the abbreviation CVSS mean?

The letters stand for the words: Common Vulnerability Scoring System. That means something like a general vulnerability rating system. Here, the weak points found are evaluated from various points of view. These elements are weighted against each other so that a standardized number between 0 and 10 is obtained at the end.

For what do you need such a rating system?

A rating system that provides us with a standardized number allows us to evaluate different weak points abstractly and derive follow-up actions from them. The focus here is on standardizing the handling of these weak points. As a result, you can define actions based on the value ranges. Here I mean processes in the value creation sources that are affected by this weak point.

What is the basic structure of this assessment?

In principle, CVSS can be described so that the probability and the maximum possible damage are related using predefined factors. The basic formula for this is:risk = probability of occurrence x damage

https://youtu.be/CZrS_UFT37Y

The Basic Values from 0..10

The evaluation in the CVSS is based on various criteria and is called “metrics”. For each metric, one or more values selected from a firmly defined selection option. This selection then results in a value between 0.0 and 10.0. Where 0 is the lowest and 10 is the highest risk value. The entire range of values ​​is then subdivided into groups and are labelled “None “, “Low “, “Medium “, “High “, and “Critical “. These metrics are divided into three areas that are weighted differently from one another. These are the areas “Basic Metrics”, “Temporal Metrics”, and “Environmental Metrics”. Here, different aspects are queried in each area, which must be assigned a single value. The weighting among each other and the subsequent composition of the three group values ​​gives the final result.

However, all component values ​​that lead to this result are always supplied. This behaviour ensures that there is transparency at all times about how these values ​​originally came about. Next, the three sub-areas will be explained individually in detail.

The Basic Metrics

The basic metrics form the foundation of this rating system. The aim is to record the technical details of the vulnerability that will not change over time. You can imagine it to be an assessment that is independent of other changing elements. Different parties can carry out the calculation of the base value. It can be done by the discoverer, the manufacturer of the project or product concerned or by a party (CERT) charged with eliminating this weak point. One can imagine that, based on this initial decision, the value itself will turn out differently since the individual groups pursue different goals.

necessary requirements:

The base value evaluates the prerequisites that are necessary for a successful attack via this security gap. This is, for example, the distinction between whether a user account must be available on the target system for an attack that is used in the course of the attack or whether the system can be compromised without the knowledge about a system user. It also plays a significant role in whether a system is vulnerable over the Internet or whether physical access to the affected component is required.

Complexity of the attack:

The base value should also reflect how complex the attack is to carry out. In this case, the complexity relates to the necessary technical steps and includes assessing whether the interaction with a regular user is essential. Is it sufficient to encourage any user to interact, or does this user have to belong to a specific system group (e.g. administrator)? At this point, it is already evident that the assessment of a new vulnerability requires exact knowledge of this vulnerability and the systems concerned. The correct classification is not a trivial process.

Assessment of the damage:

The basic metrics also take into account the damage that this attack could cause to the affected component. This means the possibilities to extract the data from the system, manipulate it, or completely prevent the system’s use. One speaks here of the three areas;

• Confidentiality
• Integrity
• Availability

However, you have to be careful here concerning the weighting of these possibilities. In one case, it can be worse when data is stolen than it is changed. In another case, the unusability of a component can be the worst damage to be assumed.

Scope-Metric:

The scope metric has also been available since CVSS version 3.0. This metric looks at the effects of an affected component on other system components. For example, one can imagine that a compromised element in a virtualized environment enables access to the carrier system. A successful change of this scope represents a greater risk for the overall system and is therefore also evaluated using this factor. This point alone clearly shows that the interpretation of the values also requires adjusting to one’s situation. And so we come to the “temporal” and “environment” metrics.

The Temporal Metrics

The time-dependent components of the vulnerability assessment are brought together in the group of temporal metrics. The peculiarity at this point is that the base value can be reduced by the temporal components only. The initial rating is intended to represent the worst-case scenario.

This has both advantages and disadvantages if you bear in mind that it is during the initial assessment of a vulnerability that can give very different interests. At this point, there are two things that need to be highlighted;

_1) Which factors influence the temporal metrics? _

The elements that change over time influence the “Temporal Metrics”.

On the one hand, this refers to changes concerning the availability of tools that support the exploitation of the vulnerability. These can be exploits or step-by-step instructions. A distinction must be made whether a chess point is theoretical or whether a manufacturer has officially confirmed it. All of these events change the base value.

2) Influence of the initial evaluation?

The influence on the initial evaluation comes about through external framework conditions. These take place over an undefined time frame and are not relevant for the actual basic assessment. Even if an exploit is already in circulation during the base values survey, this knowledge will not be included in the primary assessment. However, the base value can only be reduced by the temporal metrics. This approach takes a little getting used to and is often the subject of criticism. The reason why you decided on it is understandable from the theoretical point of view. The base value is intended to denote the most excellent possible damage.

And this is where a conflict arises. The person or group who has found a security gap tries to set the base value as high as possible. A highly critical loophole is better to sell and better exploited in the media. The reputation of the person/group who found this gap increases as a result. The affected company or the affected project is interested in exactly the opposite assessment. It, therefore, depends on who finds the security gap, how the recycling should take place and by which body the first evaluation is carried out. The only offsetting component is the Environmental Metrics.

The Environmental Metrics

In the case of environmental metrics, the own system landscape is set in relation to the security gap. This means that the evaluation is adjusted based on the real situation. In contrast to Temporal Metrics, Environmental Metrics can correct the base value in both directions. The environment can therefore lead to a higher classification and must also be constantly adapted to your own changes. The combination now gives rise to purely practical questions. Let’s assume that there is a patch from the manufacturer for a security hole. The mere presence of this modification leads to a lowering of the total value in the Temporal Metrics. However, as long as the patch has not been activated in your own systems, the overall value must be drastically corrected upwards again via the Environmental Metrics. Why did I say at this point that the value has to be increased drastically? As soon as a patch is available, this patch can be used to better understand the security gap and its effects. The attacker has more and more detailed information that can be used. This reduces the resistance of the not yet hardened systems.

The Final Score

At the end of an evaluation, the final score is obtained, calculated from the three previously mentioned values. At this point, I will not explain the calculation details; I have a separate post on this. The resulting value is then assigned to a value group. But there is one more point that is very important to me personally. In many cases, I see that the final score is simply carried over. The individual adjustments utilizing the environmental score do not take place. Instead, the value one to one is adopted as the risk assessment. In many cases, this behaviour leads to a dangerous evaluation which is incorrect for the overall system concerned.

Conclusion

We come to the management summary. With the CVSS we have a value system for evaluating security gaps in software. Since there are no alternatives, the system has been in use worldwide for over ten years and is constantly being developed, it is a defacto standard. The evaluation consists of three components. First, there is the basic score, which is there to depict a purely technical worst-case scenario. The second component is the evaluation of the time-dependent corrections based on external influences. This is about the evaluation of whether there are further findings, tools or patches for this security gap. The peculiarity of this point is that the base score can only be reduced with this value. The third component is the assessment of your own system environment with regard to this weak point. With this consideration, the security gap is adjusted in relation to the real situation on site. This value can lower or higher the base rating. Last but not least, an overall evaluation is made from these three values, which results in a number between 0.0 to 10.0.

This final value can be used to control your own actions to defend against the security gap in the overall context. At first glance, everything is quite abstract. It takes some practice to get a feel for the figures it contains.

The CVSS can only develop its full effect if it deals more deeply with this matter within your own system.

I will explain the underlying calculations in detail in further posts.

]]>DevSecOpshttps://svenruppert.com/posts/pattern-from-the-practical-life-of-a-software-developer/Tue, 09 Mar 2021 21:28:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/pattern-from-the-practical-life-of-a-software-developer/Builder-Pattern The book from the “gang of four” is part of the essential reading in just about every computer science branch. The basic patterns are described and grouped to get a good start on the topic of design patterns. But how does it look later in use? Here we will take a closer look at one pattern and expand it.<![CDATA[

Builder-Pattern

The book from the “gang of four” is part of the essential reading in just about every computer science branch. The basic patterns are described and grouped to get a good start on the topic of design patterns. But how does it look later in use?
Here we will take a closer look at one pattern and expand it.

---

https://youtu.be/n7X_XT55-jw

---

The Pattern - Builder

The builder pattern is currently enjoying increasing popularity as it allows you to build a fluent API.
It is also lovely that an IDE can generate this pattern quite quickly. But how about using this design pattern in daily life?

The basic builder pattern

Let’s start with the basic pattern, the initial version with which we have already gained all our experience.
For example, I’ll take aCar class with theEngine andList< Wheels> attributes. A car’s description is certainly not very precise, but it is enough to demonstrate some specific builder-pattern behaviours.

Now let’s start with theCar class.

javaCopy

publicclassCar{privateEngineengine;privateListwheelList;//SNIPP}

At this point, I leave out the get and set methods in this listing. If you generate a builder for this, you get something like the following.

xmlCopy

public static final class Builder { private Engine engine; private List<Wheel> wheelList; private Builder() { } public Builder withEngine(Engine engine) { this.engine = engine; return this; } public Builder withWheelList(List<Wheel> wheelList) { this.wheelList = wheelList; return this; } public Car build() { return new Car(this); } }

Here the builder is implemented as a static inner class. The constructor of the “Car” class has also been modified.

javaCopy

privateCar(Builderbuilder){setEngine(builder.engine);wheelList=builder.wheelList;}

On the one hand, there has been a change frompublic toprivate , and on the other hand, an instance of the builder has been added as a method parameter.

javaCopy

Carcar=Car.newBuilder().withEngine(engine).withWheelList(wheels)

An example - the car

If you now work with the Builder Pattern, you get to the point where you have to build complex objects. Let us now extend our example by looking at the remaining attributes of theCar class.

xmlCopy

public class Car { private Engine engine; private List<Wheel> wheelList;}public class Engine { private int power; private int type;}public class Wheel { private int size; private int type; private int colour;}

Now you can have a corresponding builder generated for each of these classes. If you stick to the basic pattern, it looks something like this for the classWheel :

javaCopy

publicstaticfinalclassBuilder{privateintsize;privateinttype;privateintcolour;privateBuilder(){}publicBuilderwithSize(intsize){this.size=size;returnthis;}publicBuilderwithType(inttype){this.type=type;returnthis;}publicBuilderwithColour(intcolour){this.colour=colour;returnthis;}publicWheelbuild(){returnnewWheel(this);}}

But what does it look like if you want to create an instance of the classCar? For each complex attribute ofCar , we will create an instance using the builder. The resulting source code is quite extensive; at first glance, there was no reduction in volume or complexity.

xmlCopy

public class Main { public static void main(String[] args) { Engine engine = Engine.newBuilder().withPower(100).withType(5).build(); Wheel wheel1 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); Wheel wheel2 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); Wheel wheel3 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); List<Wheel> wheels = new ArrayList<>(); wheels.add(wheel1); wheels.add(wheel2); wheels.add(wheel3); Car car = Car.newBuilder() .withEngine(engine) .withWheelList(wheels) .build(); System.out.println("car = " + car); }}

This source code is not very nice and by no means compact. So how can you adapt the builder pattern here so that on the one hand you have to write as little as possible by the builder himself and on the other hand you get more comfort when using it?

WheelListBuilder

Let’s take a little detour first. To be able to raise all potentials, we have to make the source text homogeneous. This strategy enables us to recognize patterns more easily. In our example, the creation of theList is to be outsourced to a builder, aWheelListBuilder.

xmlCopy

public class WheelListBuilder { public static WheelListBuilder newBuilder(){ return new WheelListBuilder(); } private WheelListBuilder() {} private List<Wheel> wheelList; public WheelListBuilder withNewList(){ this.wheelList = new ArrayList<>(); return this; } public WheelListBuilder withList(List wheelList){ this.wheelList = wheelList; return this; } public WheelListBuilder addWheel(Wheel wheel){ this.wheelList.add(wheel); return this; } public List<Wheel> build(){ //test if there are 4 instances.... return this.wheelList; }}

Now our example from before looks like this:

xmlCopy

public class Main { public static void main(String[] args) { Engine engine = Engine.newBuilder().withPower(100).withType(5).build(); Wheel wheel1 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); Wheel wheel2 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); Wheel wheel3 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); List<Wheel> wheelList = WheelListBuilder.newBuilder() .withNewList() .addWheel(wheel1) .addWheel(wheel2) .addWheel(wheel3) .build();//more robust if you add tests at build() Car car = Car.newBuilder() .withEngine(engine) .withWheelList(wheelList) .build(); System.out.println("car = " + car); }}

Next, we connect theWheel class builder and theWheelListBuilder class. The goal is to get a fluent API so that we don’t create the instances of theWheel class individually and then use theaddWheel(Wheel w) method toWheelListBuilder need to add. It should then look like this for the developer in use:

javaCopy

Listwheels=wheelListBuilder.addWheel().withType(1).withSize(2).withColour(2).addWheelToList().addWheel().withType(1).withSize(2).withColour(2).addWheelToList().addWheel().withType(1).withSize(2).withColour(2).addWheelToList().addWheel().withType(1).withSize(2).withColour(2).addWheelToList().build();

So what happens here is the following: As soon as theaddWheel() method is called, a new instance of the classWheelBuilder should be returned. TheaddWheelToList() method creates the representative of theWheel class and adds it to the list. To do that, you have to modify the two builders involved. TheaddWheelToList() method is added to theWheelBuilder side. This adds the instance of theWheel class to theWheelListBuilder and returns the instance of theWheelListBuilder class.

javaCopy

privateWheelListBuilderwheelListBuilder;publicWheelListBuilderaddWheelToList(){this.wheelListBuilder.addWheel(this.build());returnthis.wheelListBuilder;}

On the side of theWheelListBuilder class, only the methodaddWheel() is added.

javaCopy

publicWheel.BuilderaddWheel(){Wheel.Builderbuilder=Wheel.newBuilder();builder.withWheelListBuilder(this);returnbuilder;}

If we now transfer this to the other builders, we come to a pretty good result:

javaCopy

Carcar=Car.newBuilder().addEngine().withPower(100).withType(5).done().addWheels().addWheel().withType(1).withSize(2).withColour(2).addWheelToList().addWheel().withType(1).withSize(2).withColour(2).addWheelToList().addWheel().withType(1).withSize(2).withColour(2).addWheelToList().done().build();

The NestedBuilder

So far, the builders have been modified individually by hand. However, this can be implemented generically quite easily since it is just a tree of builders.

Every builder knows his children and his father. The implementations required for this can be found in theNestedBuilder class. It is assumed here that the methods for setting attributes always begin with the prefixwith. Since this seems to be the case with most generators for builders, no manual adjustment is necessary here. The methoddone() sets the result of his methodbuild() on his father. The call is made using reflection. With this, a father knows the authority of the child. At this point, I assume that the name of the attribute is the same as the class name. We will see later how this can be achieved with different attribute names. The methodwithParentBuilder(..) enables the father to announce himself to his child. We have a bidirectional connection now.

xmlCopy

public abstract class NestedBuilder<T,V> { public T done() { Class<?> parentClass = parent.getClass(); try { V build = this.build(); String methodname = "with" + build.getClass().getSimpleName(); Method method = parentClass.getDeclaredMethod(methodname, build.getClass()); method.invoke(parent, build); } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) { e.printStackTrace(); } return parent; } public abstract V build(); protected T parent; public<PextendsNestedBuilder<T,V>> P withParentBuilder(T parent) { this.parent = parent; return (P) this; }}

Now the specific methods for connecting with the children can be added to a father. There is no need to derive fromNestedBuilder.

javaCopy

publicclassParent{privateKidAkidA;privateKidBkidB;//snipp.....publicstaticfinalclassBuilder{privateKidAkidA;privateKidBkidB;//snipp.....// to add manuallyprivateKidA.BuilderbuilderKidA=KidA.newBuilder().withParentBuilder(this);privateKidB.BuilderbuilderKidB=KidB.newBuilder().withParentBuilder(this);publicKidA.BuilderaddKidA(){returnthis.builderKidA;}publicKidB.BuilderaddKidB(){returnthis.builderKidB;}//---------publicParentbuild(){returnnewParent(this);}}}

And with the children, it looks like this: Here, you only have to derive fromNestedBuilder.

javaCopy

publicclassKidA{privateStringnote;//snipp.....publicstaticfinalclassBuilderextendsNestedBuilder<Parent.Builder,KidA>{//snipp.....}}

The use is then very compact, as shown in the previous example.

javaCopy

publicclassMain{publicstaticvoidmain(String[]args){Parentbuild=Parent.newBuilder().addKidA().withNote("A").done().addKidB().withNote("B").done().build();System.out.println("build = "+build);}}

Any combination is, of course, also possible. This means that a proxy can be a father and child at the same time. Nothing stands in the way of building complex structures.

javaCopy

publicclassMain{publicstaticvoidmain(String[]args){Parentbuild=Parent.newBuilder().addKidA().withNote("A").addKidB().withNote("B").done().done().build();System.out.println("build = "+build);}}

Happy Coding

]]>Javahttps://svenruppert.com/posts/delegation-versus-inheritance-in-graphical-user-interfaces/Thu, 18 Feb 2021 17:26:16 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/delegation-versus-inheritance-in-graphical-user-interfaces/ Intro In this article, we will look at the difference between the inheritance and delegation concepts. Or, to put it better, why I prefer delegation and why I want to emphasize this rarely-used feature in Java.<![CDATA[

Intro

In this article, we will look at the difference between the inheritance and delegation concepts. Or, to put it better, why I prefer delegation and why I want to emphasize this rarely-used feature in Java.

The Challenge

The challenge we face today is quite common in the field of graphic user interfaces like desktop- or web-apps. Java is widely used as the development language for both worlds, and it does not matter if we are in the classic swing, JavaFX, or the field of web frameworks like Vaadin. Explicitly, I’ve opted for a pseudo-class model in core Java, as I’d like to look at the design patterns here without any technical details.

The goal is to create a custom component that consists of a text input field and a button. Both elements should be displayed next to each other, i.e. in a horizontal layout. The respective components have no function in this example. I want to be here exclusively to work towards the differences between inheritance and delegation.

To lazy to read? Check-out my Youtube Version!

https://youtu.be/xgViS-wlH7Q

The Base Class Model

Mostly, there are the respective essential components in a framework. In our case, it is a TextField, a button, and a horizontal or vertical layout. However, all of these components are embedded in an inheritance structure. In our case, I chose the following construction. Each component corresponds to the Component interface, for which there is an abstract implementation called AbstractComponent.

The class AbstractComponent contains framework-specific and technologically-based implementations. The **Button, **as well as theTextField , extend the classAbstractComponent. Layouts are usually separate and, therefore, a specialized group of components that leads in our case to an abstract class namedLayout , which inherits from the classAbstractComponent.

In this abstract class, there are layout-specific implementations that are the same for all sorts of layouts. The implementationsHorizontalLayout andVerticalLayout based on this. Altogether, this is already a quite complex initial model.

Inheritance — First Version

In the first version, I show a solution that I have often seen in projects. As a basis for a custom component, a base component from the framework is used as a parent. The direct inheritance from a layout is often used to structure all other internally child components on the screen. Inside the constructor, the internally required elements are generated and added to the inherited layout structure.

javaCopy

publicclassInputComponentextendsHorizontalLayout// Layout is abstractimplementsHasLogger{privatebuttonbutton=newButton();privateTextFieldtextField=newTextField();publicInputComponent(){addComponent(textfield);addComponent(button);}publicvoidclick(){button.click();}publicvoidsetText(Stringtext){textField.setText(text);}publicStringgetText(){returntextField.getText();}}

If you now look at how the component will behave during later use, it becomes visible that a derivation from a fundamental component brings its pitfalls with it.

What exactly happened here? If an instance of the custom componentInputComponent is now used, it can be viewed as a layout. But that is not the case here anymore; on the contrary, it is even wrong. All methods inherited from the layout implementation are also public available with this component. But you wanted to achieve something else. First of all, we wanted to reuse the existing code, provided in the component implementationHorizontalLayout.

On the other hand, you want a component that externally delegates only the methods for the necessary interaction, needed for the custom behaviour. In this case, the public methods from the Button and theTextField used symbolically. Besides, this component is tied to visual design that leads to possible interactions that are not part of the domain-specific behaviour of this component. This technical debt should be avoided as much as possible.

In practical words, general methods from the implementation of theHorizontalLayout are made visible to the outside. If somebody uses exactly these methods, and later on the parent becomes aVerticalLayout , the source code can not compile without further corrections.

javaCopy

publicclassMainM01implementsHasLogger{publicstaticvoidmain(String[]args){varinputComponent=newInputComponent();inputComponent.setText("Hello Text M01");inputComponent.click();// critical thingsinputComponent.doSomethingLayoutSpecific();inputComponent.horizontalSpecific();inputComponent.doFrameworkSpecificThings();}}

Inheritance — Second Version

The custom component has to fit into the already existing component hierarchy from the framework. A place must be found inside the inheritance to start from; otherwise, the custom component cannot be used. But at the same time, we do not want to own specific implementation details, and neither the effort to implement basic technical requirements based on the framework needs. The point from which you split up the inheritance must be used wisely.

Please assume that the classAbstractComponent is what we are looking for as a start point.
If you derive your class from it, so you certainly have the essential features that you would like to have as a user of the framework. However, this abstraction mostly associated with the fact that also framework-specific things are to be considered. This abstract class is an internally used, fundamental element. Starting with this internal abstract class very likely leads to the need to implement internal and technical related methods. As an example, the method signature with the namedoFrameworkSpecificThings() has been created and implemented with just a log message.

javaCopy

publicclassInputComponentextendsAbstractComponentimplementsHasLogger{privatebuttonbutton=newButton();privateTextFieldtextField=newTextField();publicInputComponent(){varlayout=newHorizontalLayout();layout.addComponent(textfield);layout.addComponent(button);addComponent(layout);}publicvoidclick(){button.click();}publicvoidsetText(Stringtext){textField.setText(text);}publicStringgetText(){returntextField.getText();}// to deep into the framework for EndUserpublicvoiddoFrameworkSpecificThings(){logger().info("doFrameworkSpecificThings -"+this.getClass().getSimpleName());}}

In use, such a component is already a little less dangerous. Only the internal methods that are visible on other components are accessible on this component.

javaCopy

publicclassMainM02implementsHasLogger{publicstaticvoidmain(String[]args){varinputComponent=newInputComponent();inputComponent.setText("Hello Text M02");inputComponent.click();// critical thingsinputComponent.doFrameworkSpecificThings();}}

But I am not happy with this solution yet. Very often, there is no requirement for new components on the technical side. Instead, they are compositions of already existing essential elements, composed in a professional, domain-specific context.

Composition — My Favorite

So what can you do at this point? The beautiful thing about the solution is that you can use it to put a wrapper around already existing components, which have been generated by inheritance. One solution may be to create a composite of typeT.** Composite**

This class serves as an envelope for the compositions of the required components. This class can then even itself inherit from the interface Component, so those technical methods of the abstract implementation not repeated or released to the outside. The typeT itself is the type to be used as the external component that holds in the composition. In our case, it is the horizontal layout. With the methodgetComponent(), you can access this instance if necessary.

javaCopy

publicfinalclassInputComponentextendsCompositeimplementsHasLogger{privatebuttonbutton=newButton();privateTextFieldtextField=newTextField();publicInputComponent(){super(newHorizontalLayout());getComponent().addComponent(textfield);getComponent().addComponent(button);}publicvoidclick(){button.click();}publicvoidsetText(Stringtext){textField.setText(text);}publicStringgetText(){returntextField.getText();}}

Seen in this way, it is a neutral shell, but it will behave towards the outside as a minimal component since the minimum contract via theComponent interface. Again, only the methods by delegation to the outside are made visible, which explicitly provided. Use is, therefore, harmless.

javaCopy

publicclassMainSolution{publicstaticvoidmain(String[]args){varinputComponent=newInputComponent();inputComponent.setText("Hello Text M03");inputComponent.click();}}

Targeted Inheritance

Let’s conclude with what I believe is rarely used Java feature at the class level. The speech is about the keyword final.

To prevent an unintentional derivation, I recommend the targeted use of final classes. Thus, from this point, the unfavourable inheritance on the composition level is troublesome. Understandably, in most frameworks, no use of it was made. After all, you want to allow the user of the componentButton to offer a specialized version. But at the beginning of your abstraction level, you can very well use it.

Conclusion

At this point, we have seen how you can achieve a more robust variant of a composition by delegation rather than inheritance. You can also use this if you are confronted with legacy source codes with this anti-pattern. It’s not always possible to clean up everything or change it to the last detail. But I hope this has given an incentive to approach this situation.

The source code for this example can be found onGitHub.

Cheers Sven!

]]>Javahttps://svenruppert.com/posts/a-challenge-of-the-software-distribution/Sun, 14 Feb 2021 14:03:57 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/a-challenge-of-the-software-distribution/ The four factors that are working against us Software development is more and more dependent on Dependencies and the frequency of deployments is increasing. Both trends together are pushing themselves higher. Another element that turns the delivery of software into a network bottleneck is the usage of compounded artefacts. And the last trend that is working against us, is the exploding amount of edges or better-called edge nodes.All four trends together are a challenge for the infrastructure.But what we could do against it?<![CDATA[

The four factors that are working against us

Software development is more and more dependent on Dependencies and the frequency of deployments is increasing. Both trends together are pushing themselves higher. Another element that turns the delivery of software into a network bottleneck is the usage of compounded artefacts. And the last trend that is working against us, is the exploding amount of edges or better-called edge nodes.All four trends together are a challenge for the infrastructure.But what we could do against it?

https://youtu.be/VSLZ4Q6ELEk

Edge-Computing

Before we look at the acceleration strategies I will explain a bit the term “Edge” or better “Edge-Computing” because this is often used in this context.

What is Edge or better edge computing?

The principle of edge computing states that data processing takes place at the Edge of the network. Which device is ultimately responsible for processing the data can differ depending on the application and the implementation of the concept.
An edge device is a device on the network periphery that generates, processes or forwards data itself. Examples of edge devices are smartphones, autonomous vehicles, sensors or IoT devices such as fire alarms.
An edge gateway is installed between the edge device and the network. It receives data from edge devices that do not have to be processed in real-time, processes specific data locally or selectively, sends the data to other services or central data centers. Edge gateways have wireless or wired interfaces to the edge devices and the communication networks for private or public clouds.

Pros of Edge Computing

The data processing takes place in the vicinity of the data source, minimising transmission and response times. Communication is possible almost in real-time. Simultaneously, the data throughput and the bandwidth usage reduction in the network, since only specific data that are not to be processed locally need to be transmitted to central data centres. Many functions can also be maintained even if the network or parts of the network fail—the performance of edge computing scales by providing more intelligent devices at the network periphery.

Cons of Edge Computing

Edge computing offers more security due to the locally limited data storage, but this is only the case if appropriate security concepts are available for the decentralised devices, due to the heterogeneity and many different devices, the effort involved in implementing the security concepts increases.

Fog Computing

Edge computing and fog computing are both decentralised data processing concepts. Fog Computing inserts another layer with the so-called Fog Nodes between the edge devices and the cloud. These are small, local data centres in the access areas of the cloud. These fog nodes collect the data from the edge devices. You select the data to be processed locally or decentrally and forward it to central servers or process it directly yourself.
Selecting the best of both worlds means we are combining both principles of Edge- and Fog-Computing.

What are the acceleration options for SW Distribution?

There are different strategies to scale the distribution of binaries, and every solution suits a specific use-case. We will not have a few on cloud solutions only because companies are operating worldwide and have to deal with different governmental regulations and restrictions. Additionally, to these restrictions, I want to highlight the need for hybrid solutions as well. Hybrid solutions are including on-prem resources as well as air gaped infrastructure used for high-security environments.

a) Custom Solution based on replication or scaling servers

One possibility to scale inside your network/architecture is scaling hardware and working with direct replication. Implementing this by yourself will most-likely consume a higher budget of workforce, knowledge, time and money based on the fact that this is not a trivial project. At the same time, this approach is bound into the borders of the infrastructure you have access to.

b) P2P Networks

Peer to Peer networks is based on equal nodes that are sharing the binaries between all nodes.The peer to peer approach implies that you will have a bunch of copies of your files. If you are downloading a file from the network, all nodes can serve parts independently. This approach of splitting up files and delivering from different nodes simultaneously to the requesting node leads to constant and efficient network usage and reduced download times.

c) CDN - Content Delivery Network

CDN’s are optimised to deliver large files over regions. The network itself is build-out of a huge number of nodes that are caching files for regional delivery. With this strategy, the original server will not be overloaded.

javaCopy

_Checkoutonmy**Youtube**Channelthevideowiththetitle"**DevSecOps - the Low hanging fruits** ".ThisvideodescribesthebalancebetweenwritingthecodeitselforaddingadependencyineachCloud-NativeApplayer.Thequestionis,whatdoesthismeanforDevSecOps?_

JFrog Solution

With the three mentioned techniques you can build up huge and powerful architecture that fit´s to your needs. But the integration of all these technologies and implementing products is not easy. We faced this challenge as well and over the years we found solutions that we integrated into a single DevSecOps Platform called “The JFrog Platform “. I don´t want to give an overview of all components, for this check out my youtube channel. Here I want to focus on the components that are responsible for the Distribution of the binaries only.

JFrog Distribution

With theJFrog Distribution, the knowledge about the content of the repositories and the corresponding metadata is used to provide a replication strategy. The replication solution is designed for internal and external repositories to bring the binaries all the way down to the place where it is needed. The infrastructure can be built in a hybrid model, including on-prem and cloud nodes.Even air-gapped solutions are possible with import/export mechanisms. In this scenario, we are focussing on a scalable caching mechanism that is optimised for reads.

What is a Release Bundle?

A Release bundle is a composition of binaries. These binaries can be of different types, like maven, Debian or Docker. TheRelease Bundle can be seen as a Bills Of Materials (BOM).The content and well as the Release Bundles itself are immutable. This immutability makes it possible to implement efficient caching and replication mechanisms across different networks and regions.

What is an Edge Node in this context?

An Edge Node in our context is a node that will provide the functionality of a read-only Artifactory.With this Edge Node, the delivery process is optimised, and we will see that replication is done in a transactional way. The difference to the original meaning of an Edge Node is that this instance is not the consuming or producing element. This can be seen as a Fog-Node, that is the first layer above the real edge nodes layer.

P2P Download

TheP2P solution focuses on environments that need to handle download bursts inside the same network or region.This download bursts could be scenarios like “updating a server farm” or “updating a Microservice Mesh”. The usage is unidirectional, which means that the consumer is not updating from their side. They are just waiting for a new version and all consumer updating at the same time.This behaviour is a perfect case for the P2P solution. Artifactory, or an Edge Node in the same network or region, is influencing an update of all P2P Nodes with a new version of a binary. The consumer itself will request the binary from the P2P node and not from the Artifactory instance anymore.The responsible Artifactory instance manages the P2P nodes, which leads to zero maintenance on the user side. Have in mind, that the RBAC is active at the P2P nodes as well.

CDN Distribution

The CDN Solution is optimised to deliver binaries to different parts of the world. We have it in two flavours. One is for the public and mostly used to distribute SDK’s, Drivers or other free available binaries. The other flavour is focussing on the private distribution.Whatever solution you are using, the RBAC defined inside the Access Module is respected, including solutions with Authentication and Authorisation and unique links including Access Tokens.

Conclusion

Ok, it is time for the conclusion.What we discussed today;
With the increasing amount of dependencies, a higher frequency of deployments and the constantly growing number of applications and edge-nodes, we are facing scalability challenges.
We had a look at three ways you could go to increase your delivery speed.The discussed solution based on

a) JFrog Distribution helps you build up a strong replication strategy inside your hybrid infrastructure to speed up the development cycle.
b) JFrog P2P that will allow you to handle massive download bursts inside a network or region. This solution fits tasks that need to distribute binaries to a high number of consumers concurrently during download bursts.
c) JFrog CDN to deliver binaries worldwide into regional data centres to make the experience for the consumer as best as possible.

All this is bundled into the JFrog DevSecOps Platform.

Cheers Sven

]]>DevSecOpshttps://svenruppert.com/posts/devsecops-be-independent-again/Fri, 12 Feb 2021 18:06:03 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/devsecops-be-independent-again/ What do the effects the news of the last few months can have to do with risk management and the presumption of storage, and why is it an elementary component of DevSecOps?<![CDATA[

What do the effects the news of the last few months can have to do with risk management and the presumption of storage, and why is it an elementary component of DevSecOps?

---

If you want to see this Post as a video, check-out the following from me Youtube Channel

https://youtu.be/nXxOXgRY5_s

---

What Has Happened So Far

Again and again, changes have happened that set things in motion that were considered to have been set. In some cases, services or products have been freely available for many years, or the type of restriction has not changed. I am taking one of the last changes as an occasion to show the resulting behavior and to formulate solutions that help you deal with it.

In software development, repositories are one of the central elements that enable you to efficiently deal with the abundance of dependencies in a software environment. A wide variety of types and associated technologies have evolved over the decades. But there is a common approach mostly resulted in a global central authority that is seen as an essential reference.

**I examined the topic of the repository from a generic point of view in a little more detail on youtube. **

https://youtu.be/Wk-rlZ904to

As an example, I would like to briefly show what a minimal technology stack can look like today. Java is used for the application itself, the dependencies of which are defined usingmaven. For this, we need access to maven repositories. Debian repositories [Why Debian Repos are mission-critical..] used for the operating system on which the application is running. The components that then packaged into Docker images use Docker registries, and finally, the applications orchestrated in a composition of Docker images using Kubernetes. Here alone, we are dealing with four different repository types. At this point, I have left out the need for generic repositories to provide the required tools used within the DevSecOps pipeline.

DockerHub And Its Dominance

The example that inspired me to write this article was DockerHub’s announcements. Access to this service was free, and there were no further restrictions on storage space and storage duration for freely available Docker images. This fact has led to a large number of open source projects using this repository for their purposes. Over the years a whole network of dependencies between these images has built up.

Docker Hub was in the news recently for two reasons.

Storage Restrictions

Previously, Docker images were stored indefinitely on Dockerhub. On the one hand, this meant that nobody cared about the storage space of the Docker images. On the other hand, pretty much everyone has been counting on it not to change again. Unfortunately, that has now changed. The retention period for inactive Docker images has been reduced to six months. What doesn’t sound particularly critical at first turns out to be quite uncomfortable in detail.

Download Throttling

Docker has limited the download rate to 100 pulls per six hours for anonymous users, and 200 pulls per six hours for free accounts. Number 200 sounds pretty bearable. However, it makes sense to take a more detailed look here. 200 inquiries / 6h are 200 inquiries / 360min. We’re talking about 0.55 queries/minute at a constant query rate. First, many systems do more than one build and therefore requests, every 2 minutes. Second, if the limit is reached, it can take more than half a business day to regain access. The latter is to be viewed as very critical. As a rule, limit values given per hour, which then only leads to a delay of a little less than an hour. Six hours is a different order of magnitude.

Maven and MavenCentral

If you look at the different technologies, a similar monoculture emerges in the Maven area. Here is the maven-central a singular point operated by one company. A larger company bought this company. What does this mean for the future of this repository? I don’t know. However, it is not uncommon for costs to be optimized after a takeover by another company. A legitimate question arises here; What economic advantage does the operator of such a central, free-of-charge infrastructure have?

JDKs

There have been so many structural changes here that I’m not even sure what the official name is. But there is one thing I observe with eagle eyes in projects. Different versions, platforms and providers of the JDKs result in a source of joy in LTS projects that should not be underestimated. Here, too, it is not guaranteed how long the providers will keep the respective builds of a JDK for a platform. What is planned today can be optimized tomorrow. Here, too, you should take a look at the JDKs that are not only used internally but also by customers. Who has all the installers for the JDKs in use in stock? Are these JDKs also used within your own CI route, or do you trust the availability of specific Docker images?

Moderate Independence

How can this be countered now? The answer is straightforward. You get everything you need just once and then save it in your systems. And so we are running against the efforts of the last few years. As in most other cases, moderate use of this approach is recommended. More important than ever is the sensible use of freely available resources. It can help if a stringent retention tactic is used. Not everything has to be kept indefinitely. Many elements that are held in the caches are no longer needed after a while. Sophisticated handling of repositories and the nesting of resources helps here. Unfortunately, I cannot go into too much detail here, but it can be noted in short form.

The structure of the respective repositories enables, on the one hand, to create concrete compositions and, on the other hand, to carry out very efficient maintenance. Sources must be kept in individual repositories and then merged using virtual repositories. This process can be built up so efficiently that it can even drastically reduce the number of build cycles.

DevSecOps - Risk Minimization

There is another advantage in dealing with the subject of “independence”. Because all files that are kept in their structures can be analyzed with regard to vulnerabilities and compliance, now when these elements are in one place, in a repository manager, I have a central location where I can scan them. The result is a complete dependency graph that includes the dependencies of an application but also the associated workbench. That, in turn, is one of the critical statements when you turn to the topic of DevSecOps. Security is like quality! It’s not just a tool; it’s not just a person responsible for it. It is a philosophy that has to run through the entire value chain.

Happy Coding,

Sven Ruppert

]]>DevSecOpshttps://svenruppert.com/posts/the-quick-wins-of-devsecops/Thu, 28 Jan 2021 16:56:45 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/posts/the-quick-wins-of-devsecops/Hello and welcome to my DevSecOps post. Here in Germany, it’s winter right now, and the forests are quiet. The snow slows down everything, and it’s a beautiful time to move undisturbed through the woods.<![CDATA[

Hello and welcome to my DevSecOps post. Here in Germany, it’s winter right now, and the forests are quiet. The snow slows down everything, and it’s a beautiful time to move undisturbed through the woods.

Here you can pursue your thoughts, and I had to think about a subject that customers or participants at conferences ask me repeatedly.

The question is almost always:

What are the quick wins or low hanging fruits if you want to deal more with the topic of security in software development?

And I want to answer this question rightnow!

For the lazy ones, you can see it as youtube video as well

https://www.youtube.com/embed/lNqADishl8w

Let’s start with the definition of a phrase that often used in the business world.

Make Or Buy

Even as a software developer, you will often hear this phrase during meetings with the company’s management and sales part.

The phrase is called; “Make or Buy “. Typically, we have to decide if we want to do something ourselves or spend money to buy the requested functionality. It could be less or more functionality or different so that we have to adjust ourself to use it in our context.

But as a software developer, we have to deal with the same question every day. I am talking about dependencies. Should we write the source code by ourselves or just adding the next dependencies? Who will be responsible for removing bugs, and what is the total cost of this decision? But first, let’s take a look at the make-or-buy association inside the full tech-stack.

Diff between Make / Buy on all layers.

If we are looking at all layers of a cloud-native stack to compare the value of “make” to “buy” we will see that the component “buy” is in all layers the bigger one. But first things first.

The first step is the development of the application itself.

Assuming that we are working with Java and using maven as a dependency manager, we are most likely adding more lines of code indirectly as dependency compared to the number of lines we are writing by ourselves. The dependencies are the more prominent part, and third parties develop them. We have to be carefully, and it is good advice to check these external binaries for known vulnerabilities.

We should have the same behaviour regarding compliance and license usage. The next layer will be the operating system, in our case Linux.

And again, we are adding some configuration files and the rest are existing binaries.

The result is an application running inside the operating system that is a composition of external binaries based on our configuration.

The two following layers, Docker and Kubernetes, are leading us to the same result. Until now, we are not looking at the tool-stack for the production line itself.

All programs and utilities that are directly or indirectly used under the hood called DevSecOps are some dependencies.

All layers’ dependencies are the most significant part by far.

Checking these binaries against known Vulnerabilities is the first logical step.

one time and recurring efforts for Compliance/Vulnerabilities

Comparing the effort of scanning against known Vulnerabilities and for Compliance Issues, we see a few differences.

Let’s start with the Compliance issues.

Compliance issues:

The first step will be defining what licenses are allowed at what part of the production line. This definition of allowed license includes the dependencies during the coding time and the usage of tools and runtime environments. Defining the non-critical license types should be checked by a specialised lawyer. With this list of white labelled license types, we can start using the machine to scan on a regular base the full tool stack. After the machine found a violation, we have to remove this element, and it must be replaced by another that is licensed under a white-labelled one.

Vulnerabilities:

The recurrent effort on this site is low compared to the amount of work that vulnerabilities are producing. A slightly different workflow is needed for the handling of found vulnerabilities. Without more significant preparations, the machine can do the work on a regular base as well. The identification of a vulnerability will trigger the workflow that includes human interaction. The vulnerability must be classified internally that leads to the decisions what the following action will be.

Compliance Issues: just singular points in your full-stack

There is one other difference between Compliance Issues and Vulnerabilities. If there is a compliance issue, it is a singular point inside the overall environment. Just this single part is a defect and is not influencing other elements of the environment.

Vulnerabilities: can be combined into different attack vectors.

Vulnerabilities are a bit different. They do not only exist at the point where they are located. Additionally, they can be combined with other existing vulnerabilities in any additional layer of the environment. Vulnerabilities can be combined into different attack vectors. Every possible attack vector itself must be seen and evaluated. A set of minor vulnerabilities in different layers of the application can be combined into a highly critical risk.

Vulnerabilities: timeline from found until active in the production

I want to have an eye on as next is the timeline from vulnerability is found until the fix is in production. After a vulnerability is existing in a binary, we have nearly no control over the time until this is found. It depends on the person itself if the vulnerability is reported to the creator of the binary, a commercial security service, a government or it will be sold on a darknet marketplace. But, assuming that the information is reported to the binary creator itself, it will take some time until the data is publicly available. We have no control over the duration from finding the vulnerability to the time that the information is publicly available. The next period is based on the commercial aspect of this issue.

As a consumer, we can only get the information as soon as possible is spending money.
This state of affairs is not nice, but mostly the truth.

Nevertheless, at some point, the information is consumable for us. If you are usingJFrog Xray , from the free tier, for example, you will get the information very fast.JFrog is consuming different security information resources and merging all information into a single vulnerability database. After this database is fed with new information, allJFrog Xray instances are updated. After this stage is reached, you can act.

Test-Coverage is your safety-belt; try Mutation Testing.

Until now, the only thing you can do to speed up the information flow is spending money for professional security information aggregator. But as soon as the information is consumable for you, the timer runs. It depends on your environment how fast this security fix will be up and running in production. To minimise the amount of time a full automated CI Pipeline ist one of the critical factors.

But even more critical is excellent and robust test coverage.

Good test coverage will allow you, to switch dependency versions immediately and push this change after a green test run into production. I recommend using a more substantial test coverage as pure line-coverages. The technique called “mutation test coverage” is a powerful one.

Mutation Test Coverage
If you want to know more about this on, check out my YouTube channel. I have a video that explains the theoretical part and the practical one for Java and Kotlin.

https://www.youtube.com/embed/6Vej7YEOF8g

The need for a single point that understands all repo-types
To get a picture of the full impact graph based on all known vulnerabilities, it is crucial to understand all package managers included by the dependencies. Focussing on just one layer in the tech-stack is by far not enough.

JFrog Artifactory provides information, including the vendor-specific metadata that is part of the package managers.

JFrog Xray can consume all this knowledge and can scan all binaries that are hosted inside the repositories that are managed by Artifactory.

Vulnerabilities - IDE plugin

Shift Left means that Vulnerabilities must be eliminated as early as possible inside the production pipeline. One early-stage after the concept phase is the coding itself. At the moment you start adding dependencies to your project you are possibly adding Vulnerabilities as well.

The fastest way to get feedback regarding your dependencies is theJFrog IDE Plugin. This plugin will connect your IDE to your JFrog Xray Instance. The free tier will give you access to Vulnerability scanning. The Plugin is OpenSource and available for IntelliJ, VS-Code, Eclipse,… If you need some additional features, make a feature request on GitHub or fork the Repository add your changes and make a merge request.

Try it out by yourself -JFrog Free Tier

How to use the IDE plugin?

If you add a dependency to your project, the IDE Plugin can understand this information based on the used package manager. The IDE Plugin is connected to your JFrog Xray instance and will be queried if there is a change inside your project’s dependency definition. The information provided by Xray includes the known vulnerabilities of the added dependency. If there is a fixed version of the dependency available, the new version number will be shown.

If you want to see the IDE Plugin in Action without registering

for a Free Tier, have a look at my youtube video.

https://www.youtube.com/embed/PsghzAf-ODU

Conclusion

With the JFrog Free Tier, you have the tools in your hands to practice Shift Left and pushing it into your IDE.

Create repositories for all included technologies, use Artifactory as a proxy for your binaries and let Xray scan the full stack.

With this, you have a complete impact graph based on your full-stack and the pieces of information about known Vulnerabilities as early as possible inside your production line.

You don’t have to wait until your CI Pipeline starts complaining. This will save a lot of your time.

]]>DevSecOpsJavaTDDhttps://svenruppert.com/search/Mon, 01 Jan 0001 00:00:00 +0000sven.ruppert@gmail.com (Sven Ruppert)Sven Rupperthttps://svenruppert.com/search/<![CDATA[ ]]>