[{"content":"Many example projects overload the starting point by covering too many topics at once. Routing, data access, security, forms, theme customisations, and other integrations will then be presented in a single demo. This makes it more difficult for readers to recognise the project\u0026rsquo;s actual structure.\nThis project takes a different approach. The focus is on a compact framework comprising AppShell, MainLayout, several views, a small service, its own theme, and prepared internationalisation. The aim is to make the basic building blocks of a Vaadin Flow application visible in a way that their interactions are comprehensible.\nThe source code for this project can be found at\nGitHub at: https://3g3.eu/vdn-tpl\nThis is especially useful in Vaadin Flow, because the interface is created on the server side in Java. Classes such as AppShell, MainLayout, and MainView, therefore, shape the application\u0026rsquo;s form early on. The article follows a Vaadin-centric perspective and does not primarily examine a specialist domain, but the structure of a usable flow project.\nTechnology stack and project idea # The project relies on Vaadin Flow 25, Maven, Jetty, JDK 25, classic WAR packaging, its own theme and prepared internationalisation. This combination defines the application\u0026rsquo;s technical framework.\nVaadin Flow bundles components, layouts, routing, and interactions within a single Java structure. Maven organises build, dependencies, and plugins. Jetty provides a lightweight servlet container for local operation and near-deployment execution. JDK 25 positions the project on a current Java basis, while the WAR packaging underlines its character as a classic web application.\nTheme and internationalisation add two cross-cutting aspects to the stack: visual design and language-dependent content. As a result, the technical substructure is not only executable, but already prepared for typical extensions.\nProject Structure at a Glance # The application follows the classic Maven structure with src/main/java for source code and src/main/resources for accompanying resources. This structure creates a familiar framework and separates executable code from configuration and language resources.\nWithin the Java part, responsibilities are structured to match the application structure. Entry point and global configurations form the outer frame. Layout classes define the overarching UI framework, views represent concrete pages, and service classes handle supporting logic.\nResources for internationalisation and the theme complement this. Translation files are located in the resource area, and visual adjustments are bundled in the expected places. This makes it easy to see in the repository where new pages, additional logic or design adjustments are classified.\nApplication Entry Point: AppShell # The AppShell defines the application\u0026rsquo;s global layer. This is where settings are anchored that affect not only a single page, but the browser appearance as a whole. These include theme activation, meta information, viewport information or other application-wide configurations.\nIn the project, this role is implemented in a compact class:\n/** * Typical use cases of AppShell * ✅ Viewport \u0026amp; mobile optimization * ✅ Setting metadata (SEO, security) * ✅ Favicons, touch icons * ✅ Global JavaScript snippets (analytics, monitoring) * ✅ Global CSS (e.g., corporate branding) * ✅ Selecting a theme for the entire app */ @Meta(name = \u0026#34;author\u0026#34;, content = \u0026#34;Sven Ruppert\u0026#34;) @Viewport(\u0026#34;width=device-width, initial-scale=1.0\u0026#34;) @PWA(name = \u0026#34;Project Base for Vaadin\u0026#34;, shortName = \u0026#34;Project Base\u0026#34;) @Theme(\u0026#34;my-theme\u0026#34;) @Push public class AppShell implements AppShellConfigurator { @Override public void configurePage(AppShellSettings settings) { // settings.addFavIcon(\u0026#34;icon\u0026#34;, // \u0026#34;icons/my-favicon.png\u0026#34;, // \u0026#34;32x32\u0026#34;); // // // Externes CSS // settings.addLink(\u0026#34;stylesheet\u0026#34;, // \u0026#34;https://cdn.example.com/styles/global.css\u0026#34;); // // // Externes Script // settings.addInlineWithContents( // \u0026#34;console.log(\u0026#39;Hello from AppShell!\u0026#39;);\u0026#34;, // Inline.Wrapping.AUTOMATIC); } } The annotations already show very well what kind of responsibility belongs in this class. @Meta complements global metadata, @Viewport defines behavior on mobile devices, @PWA describes basic progressive web app properties, @Theme(\u0026ldquo;my-theme\u0026rdquo;) activates the application-wide theme, and @Push unlocks server-side push communication. In this way, the AppShell bundles only the settings relevant to the browser\u0026rsquo;s overall appearance.\nIn addition, the class implements AppShellConfigurator and provides a dedicated extensibility point with configurePage(AppShellSettings settings). Even though the method does not include any active settings in its current state, the commented-out examples already show typical fields of application: Favicons, global stylesheets, or inline scripts can be registered centrally here. This means the project has already been structurally prepared to accommodate such global adaptations.\nYour job isn\u0026rsquo;t to render business content or bundle component logic. Rather, it defines the frame in which the actual surface will later move. This creates a clearly recognisable place for global decisions, rather than scattering them across views or layout classes.\nThe Basic Structure of the UI with MainLayout # The MainLayout forms the permanent structural level of the surface. It bundles navigation, the header area, and the embedding of the content area, and thus determines how individual pages are presented within the same application.\nIn the project, this task is implemented in its own layout class, which inherits directly from AppLayout:\npublic class MainLayout extends AppLayout { public MainLayout() { createHeader(); } private void createHeader() { H1 appTitle = new H1(\u0026#34;Vaadin Flow Demo\u0026#34;); SideNav views = getPrimaryNavigation(); Scroller scroller = new Scroller(views); scroller.setClassName(LumoUtility.Padding.SMALL); DrawerToggle toggle = new DrawerToggle(); H2 viewTitle = new H2(\u0026#34;Headline\u0026#34;); HorizontalLayout wrapper = new HorizontalLayout(toggle, viewTitle); wrapper.setAlignItems(FlexComponent.Alignment.CENTER); wrapper.setSpacing(false); VerticalLayout viewHeader = new VerticalLayout(wrapper); viewHeader.setPadding(false); viewHeader.setSpacing(false); addToDrawer(appTitle, scroller); addToNavbar(viewHeader); setPrimarySection(Section.DRAWER); } private SideNav getPrimaryNavigation() { SideNav sideNav = new SideNav(); sideNav.addItem(new SideNavItem(\u0026#34;Dashboard\u0026#34;, \u0026#34;/\u0026#34; + MainView.PATH, DASHBOARD.create()), new SideNavItem(\u0026#34;Youtube\u0026#34;, \u0026#34;/\u0026#34; + YoutubeView.PATH, CART.create()), new SideNavItem(\u0026#34;About\u0026#34;, \u0026#34;/\u0026#34; + AboutView.PATH, USER_HEART.create()) ); return sideNav; } } Recurring elements are centrally housed here rather than rebuilt in each view. On this basis, individual pages can concentrate on their own content, while the MainLayout provides the overarching framework. Vaadin\u0026rsquo;s AppLayout provides a suitable basis for this.\nBetween AppShell and views, the MainLayout thus serves as the connecting UI layer: global settings remain outside, concrete content is within the views, and in between lies the permanent interface structure.\nSetting up navigation and routes cleanly # The routing structure determines the paths to the individual views. In Vaadin Flow, this is done directly at the respective class via @Route. This keeps routing closely related to the visible page structure.\nIn the project, this principle is evident in the view classes. The start page uses an empty path and binds to the MainLayout at the same time:\n/** * The main view contains a text field for getting the username and a button * that shows a greeting message in a notification. */ @Route(value = MainView.PATH, layout = MainLayout.class) public class MainView extends VerticalLayout implements LocaleChangeObserver { public static final String YOUR_NAME = \u0026#34;your.name\u0026#34;; public static final String SAY_HELLO = \u0026#34;say.hello\u0026#34;; public static final String PATH = \u0026#34;\u0026#34;; private final GreetService greetService = new GreetService(); private final Button button = new Button(); private final TextField textField = new TextField(); public MainView() { button.addClickListener(e -\u0026gt; { add(new Paragraph(greetService.greet(textField.getValue()))); }); add(textField, button); } @Override public void localeChange(LocaleChangeEvent localeChangeEvent) { button.setText(getTranslation(SAY_HELLO)); textField.setLabel(getTranslation(YOUR_NAME)); } } The other pages also follow the same pattern. AboutView and YoutubeView each define their paths via their own constants and are also embedded in the MainLayout:\n@Route(value = AboutView.PATH, layout = MainLayout.class) public class AboutView extends VerticalLayout { public static final String PATH = \u0026#34;about\u0026#34;; public AboutView() { H1 title = new H1(\u0026#34;About\u0026#34;); H2 subtitle = new H2(\u0026#34;Vaadin Flow Demo Application\u0026#34;); Paragraph description = new Paragraph(\u0026#34;This is a demo application built with Vaadin Flow \u0026#34; + \u0026#34;framework to showcase various UI components and features.\u0026#34;); Paragraph version = new Paragraph(\u0026#34;Version: 1.0.0\u0026#34;); Paragraph author = new Paragraph(\u0026#34;Created by: Sven Ruppert\u0026#34;); Paragraph bio = new Paragraph(\u0026#34;\u0026#34;\u0026#34; Sven Ruppert has been involved in software development for more than 20 years. \\ As developer advocate he is constantly looking for innovations in software development. \\ He is speaking internationally at conferences and has authored numerous technical articles and books.\u0026#34;\u0026#34;\u0026#34;); Paragraph homepage = new Paragraph( new Paragraph(\u0026#34;Visit my website: \u0026#34;), new Anchor(\u0026#34;https://www.svenruppert.com\u0026#34;, \u0026#34;www.svenruppert.com\u0026#34;, BLANK)); Image vaadinLogo = new Image(\u0026#34;images/vaadin-logo.png\u0026#34;, \u0026#34;Vaadin Logo\u0026#34;); vaadinLogo.setWidth(\u0026#34;200px\u0026#34;); setSpacing(true); setPadding(true); setAlignItems(Alignment.CENTER); add(title, subtitle, description, version, author, bio, homepage, vaadinLogo); } } These excerpts show several important points. First, routing is defined directly where the respective page is implemented. Secondly, the layout = MainLayout.class assignment makes it clear on which surface frame the view will appear later. Third, the paths are not distributed as magic strings in many places, but bundled via PATH constants at the classes themselves.\nRoutes do not only take on a technical function. They organise the application into accessible areas and make it visible how pages are named and separated from each other. In conjunction with the MainLayout, this creates the navigation structure that users use to switch between individual views.\nIt is important to distinguish between routing and navigation: Routing defines accessibility, navigation defines its representation in the interface.\nThe First View: MainView as a Minimal Interaction Example # With MainView, the application can be experienced as a concrete page on first launch. It shows a simple interaction pattern of input, action, and visible reaction, making the basic idea of server-side UI programming in Vaadin immediately tangible.\nIn the project, this first view is deliberately kept compact:\n/** * The main view contains a text field for getting the username and a button * that shows a greeting message in a notification. */ @Route(value = MainView.PATH, layout = MainLayout.class) public class MainView extends VerticalLayout implements LocaleChangeObserver { public static final String YOUR_NAME = \u0026#34;your.name\u0026#34;; public static final String SAY_HELLO = \u0026#34;say.hello\u0026#34;; public static final String PATH = \u0026#34;\u0026#34;; private final GreetService greetService = new GreetService(); private final Button button = new Button(); private final TextField textField = new TextField(); public MainView() { button.addClickListener(e -\u0026gt; { add(new Paragraph(greetService.greet(textField.getValue()))); }); add(textField, button); } @Override public void localeChange(LocaleChangeEvent localeChangeEvent) { button.setText(getTranslation(SAY_HELLO)); textField.setLabel(getTranslation(YOUR_NAME)); } } This class shows the basic pattern of a Vaadin view very well. The page is directly linked to the routing via @Route and is also embedded in the MainLayout. TextField and Button define two central UI components that are directly assembled in the constructor. Clicking on the button triggers a server-side action and adds a new paragraph with the result to the layout. Interface, event handling and reaction thus remain merged in a closed Java model.\nThe already integrated multilingualism is also remarkable. Through LocaleChangeObserver, the view reacts to a language change and updates identifiers such as button text and field labels directly using translation keys. In this way, the class combines interaction and internationalisation in a very small but complete example.\nThe actual text logic is no longer directly in the view, but in a small service class:\npublic class GreetService implements Serializable { public String greet(String name) { if (name == null || name.isEmpty()) { return \u0026#34;Hello anonymous user\u0026#34;; } else { return \u0026#34;Hello \u0026#34; + name; } } } It is precisely this reduction that makes it easy to see how a first interactive page in Vaadin is structured. Components, event handling, layout integration, and a small amount of outsourced logic are already intertwined without the need for additional infrastructure. At the same time, the MainView remains part of the larger application framework of Routing and MainLayout.\nSeparating UI Logic and Business Logic # With the first interactive view, the question of responsibility arises. A view describes the interface, inputs, and reactions to user actions. Supporting logic, on the other hand, can be outsourced to its own classes.\nIn the project, this role is taken over by the GreetService. Even though its task remains small, it shows the boundary between component control in the view and application-oriented logic outside the interface. This separation facilitates later expansions, because additional rules or preparation steps do not have to be accommodated directly in the view.\nThinking about internationalisation from the outset # Texts are created directly in a Vaadin application, where components are built. This is precisely why it makes sense to remove language-dependent content from views early and manage it through translation keys and resource files.\nIn the project, this is visible directly in the MainView. The class implements LocaleChangeObserver and obtains labels not as hardwired strings, but via translation keys:\n/** * The main view contains a text field for getting the username and a button * that shows a greeting message in a notification. */ @Route(value = MainView.PATH, layout = MainLayout.class) public class MainView extends VerticalLayout implements LocaleChangeObserver { public static final String YOUR_NAME = \u0026#34;your.name\u0026#34;; public static final String SAY_HELLO = \u0026#34;say.hello\u0026#34;; public static final String PATH = \u0026#34;\u0026#34;; private final GreetService greetService = new GreetService(); private final Button button = new Button(); private final TextField textField = new TextField(); public MainView() { button.addClickListener(e -\u0026gt; { add(new Paragraph(greetService.greet(textField.getValue()))); }); add(textField, button); } @Override public void localeChange(LocaleChangeEvent localeChangeEvent) { button.setText(getTranslation(SAY_HELLO)); textField.setLabel(getTranslation(YOUR_NAME)); } } The localeChange(\u0026hellip;) method, in particular, demonstrates the practical implementation of multilingualism. Instead of entering text directly when components are created, they are resolved via getTranslation(\u0026hellip;) by key. This allows the same view to display different labels depending on the active language.\nThe associated resources are located in the project at src/main/resources/vaadin-i18n. The default language is defined in translations.properties:\n**your.name=Your name\n**say.hello=Say Hello\nThere is a separate file for German translations_de.properties:\n**your.name=Your name\n**say.hello=Say hello\nThese files clearly show how the language-dependent content is extracted from the Java code. The view only knows the keys your.name and say.hello, while the actual texts are maintained in the resources. This means that the interface remains linguistically adaptable without having to rephrase component classes themselves.\nMultilingualism is already prepared in the project. Texts are not treated as fixed components of individual components, but as content that is provided depending on the active language. In conjunction with language-dependent interface updates, it becomes apparent that internationalisation not only comprises files but also describes the application\u0026rsquo;s behaviour.\nAdditional views as a blueprint for your own pages # In addition to the MainView, AboutView, and YouTubeView, the application also displays other page types. They stand for more information-oriented content and make it clear that the same structure works not only for interactive input pages, but also for simpler content pages.\nThis makes it possible to see how new views can be embedded in routing, navigation and MainLayout without introducing their own special structures. Different page types can thus be implemented within a common framework.\nBuild, Start, and Development Mode # Hands-on work with the project includes build, local startup, and development mode. Maven bundles dependencies, build steps and plugin configurations for this purpose. This is particularly relevant for Vaadin because, in addition to the Java code, themes, resources, and Vaadin-specific processing are integrated into the development process.\nJetty takes over the local operation via a lightweight servlet container. This keeps the path from the source code to the running application short. Changes to views, layouts, translations or theme rules can be checked directly in development mode.\nWAR packaging fits into this model and keeps the application in a classic web application context.\nConclusion # The project makes the central layers of a Vaadin Flow application visible in a compact form: global configuration via the AppShell, permanent UI structure in the MainLayout, routing for accessible pages, concrete views for content, an outsourced service for supporting logic, prepared internationalisation, a separate theme and a clear build and start path.\nEspecially in this compact form, it is easy to understand how these building blocks work together. The repository thus shows not only individual Vaadin techniques, but also the structure of an application whose structure remains consistently recognisable from the first entry point to the development mode.\n","date":"18 March 2026","externalUrl":null,"permalink":"/posts/a-vaadin-starter-project-with-a-clear-focus/","section":"Posts","summary":"Many example projects overload the starting point by covering too many topics at once. Routing, data access, security, forms, theme customisations, and other integrations will then be presented in a single demo. This makes it more difficult for readers to recognise the project’s actual structure.\n","title":"A Vaadin Starter Project with a Clear Focus","type":"posts"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/categories/java/","section":"Categories","summary":"","title":"Java","type":"categories"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/tags/java/","section":"Tags","summary":"","title":"Java","type":"tags"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/categories/serverside/","section":"Categories","summary":"","title":"Serverside","type":"categories"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/","section":"Sven Ruppert","summary":"","title":"Sven Ruppert","type":"page"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/categories/vaadin/","section":"Categories","summary":"","title":"Vaadin","type":"categories"},{"content":"","date":"18 March 2026","externalUrl":null,"permalink":"/tags/vaadin/","section":"Tags","summary":"","title":"Vaadin","type":"tags"},{"content":"","date":"11 March 2026","externalUrl":null,"permalink":"/tags/i18n/","section":"Tags","summary":"","title":"I18N","type":"tags"},{"content":"Modern web applications are rarely used only by users with the same language. Even internal tools often reach international teams or are used in different countries. A multilingual user interface is therefore not a luxury feature, but an important part of the user experience.\nThe open-source project URL-Shortener also benefits from a clear internationalisation strategy. The application is intended for developers, administrators, and other users who want to manage, analyse, or distribute links. To ensure that the interface remains understandable regardless of the user\u0026rsquo;s origin, the application supports multiple languages.\nThe project can be found on GitHub at the URL:https://3g3.eu/url\nThis article shows how the URL shortener UI has been enhanced with simple, robust internationalisation. The implementation is based entirely on Vaadin Flow\u0026rsquo;s capabilities and deliberately uses only a few additional helper classes to keep the architecture clear.\nInternationalisation in Vaadin Structuring Translation Resources Simplified access to translations with I18nSupport Determining the user locale Session-based voice management Language switching in the user interface Conclusion and outlook The solution has several key objectives. On the one hand, the users\u0026rsquo; language is to be automatically recognised based on the locale transmitted by the browser. In addition, it must be possible to manage multiple translations via resource bundles without creating additional administrative overhead in the application code. At the same time, the user should be able to change the language at any time during a running session. Another important goal is to keep the implementation lean and deliberate and to dispense with additional frameworks so that internationalisation can be implemented directly within the existing application without complex infrastructure.\nCurrently, the application supports three languages:\nEnglish German Finnish The choice of these languages is deliberately pragmatic. English serves as the standard language, while German and Finnish depict real-life usage scenarios from the project\u0026rsquo;s environment.\nThe focus of this article is not only on the technical implementation but also on how internationalisation can be integrated into a Java web application in a way that keeps the source code clearly structured and maintainable. To do this, we examine both the organisation of translation resources and the handling of locales within the application.\nIn the following course, it is explained step by step how Vaadin recognises and loads translation resources, how translation keys can be structured sensibly, and how the user\u0026rsquo;s language is determined based on the browser locale. In addition, it shows how simple language switching can be integrated directly into the user interface, allowing users to switch the desired language during a running session.\nThis creates an internationalisation solution that is deliberately kept simple but can be easily expanded if other languages are added later.\nInternationalisation in Vaadin # Vaadin Flow already provides basic support for internationalisation. Applications can provide translations via so-called resource bundles, which are automatically recognised and loaded by the runtime environment. In this way, user interfaces can be operated in several languages without much additional effort.\nThe central mechanism relies on translation files stored in a special directory within the project. By default, Vaadin searches for translation resources in the vaadin-i18n folder within the classpath. All files stored there are automatically recognised and are available to the application at runtime.\nThe translation files follow the familiar structure of Java Resource Bundles. A base file contains the standard translations, while additional files provide language-specific variants. The language is defined by a language suffix in the file name.\nA typical setup looks like this, for example:\nvaadin-i18n/\n** translations.properties**\n** translations_de.properties**\n** translations_fi.properties**\nThe file without a language suffix serves as the default resource. In many projects, it contains the English texts. As soon as Vaadin determines an active locale, the framework automatically attempts to load the appropriate translation file. If a language-specific variant exists, it is used. Otherwise, Vaadin falls back on the default resource.\nThis behaviour corresponds to the classic fallback mechanism of Java Resource Bundles. This means that the application remains functional even if individual translations are not yet complete.\nFor the user interface, this means that text is no longer defined directly in the source code but is referenced via unique translation keys. These keys are then resolved at runtime via the active locale.\nVaadin provides the getTranslation() method for this purpose, which can be used directly in UI components. This method is used to pass a translation key to the framework, after which Vaadin determines the appropriate text from the loaded resources.\nThus, the resource bundle system forms the basis for all other internationalisation mechanisms within the application. On this basis, additional concepts such as language switching, session-based locale management or structured translation keys can then be developed.\nStructuring Translation Resources # After looking at Vaadin\u0026rsquo;s basic internationalisation support in the previous chapter, an important question quickly arises in practice: How should translation resources be organised within a project to remain maintainable in the long term?\nIn the URL shortener project, all translations are stored in so-called resource bundles. These are located in the directory src/main/resources/vaadin-i18n/. Vaadin automatically detects this folder and loads the translation files it contains at runtime.\nThe structure of the files follows the conventions of Java Resource Bundles. A base file contains the standard translations, while additional files provide language-specific variants.\nsrc/main/resources/vaadin-i18n/\n** translations.properties**\n** translations_de.properties**\n** translations_fi.properties**\nThe translations.properties file serves as the default resource. This project contains the English texts. For other languages, additional files with a language suffix are created. The language is defined by the ISO language code in the file name.\nFor example, the German version contains the file translations_de.properties, while the Finnish version contains translations_fi.properties.\nWithin these files, the actual translations are defined via key-value pairs. Each key corresponds to a specific text in the user interface.\nmain.appTitle=URL Shortener\nmain.logout=Logout\nnav.overview=Overview\nnav.create=Create\nnav.youtube=Youtube\nnav.about=About\nThe keys themselves follow a simple naming convention. They are often built according to the pattern bereich.element. This allows texts that belong together to be logically grouped.\nThis structure makes it easier to keep track of a large number of translations. At the same time, it prevents name conflicts between different areas of the application.\nAnother advantage of this structure is that translations remain completely separate from the application code. The source code only references the keys, while the actual texts are defined in the resource bundles.\nThis allows translations to be expanded or corrected later without changes to the application code.\nThis clear separation between code and translation resources provides an important basis for further internationalisation of the application. The next step will therefore look at how these translation keys can be used comfortably within the user interface.\nSimplified access to translations with I18nSupport # Once the structure of the translation resources has been determined, another question arises in practical development: How can these translations be used as comfortably as possible within the user interface?\nVaadin basically provides the getTranslation() method, which can be used to resolve translation keys at runtime. This method is available, for example, within UI components and allows you to use a key to determine the appropriate text from the loaded resource bundles.\nIn practice, however, direct use of this method often results in recurring boilerplate code. Especially in larger user interfaces, translations have to be resolved in many different places. This quickly results in redundant calls and less readable code.\nTo make this access easier, the URL shortener project uses a small helper interface called I18nSupport. This interface encapsulates access to the translation function and provides a compact method for querying translations within the UI components.\nA simplified example of this interface looks like this:\npublic interface I18nSupport { default String tr(String key, String fallback) { return UI.getCurrent().getTranslation(key); } } This auxiliary method allows translations to be used much more compactly within the application. Instead of working directly with the Vaadin API, a single method call suffices.\nAn example from the application\u0026rsquo;s MainLayout illustrates this:\nH1 appTitle = new H1(tr(\u0026ldquo;main.appTitle\u0026rdquo;, \u0026ldquo;URL Shortener\u0026rdquo;));\nThis keeps the source code clear and the actual text completely separate from the user interface implementation.\nAnother advantage of this solution is that access to translations can be adjusted centrally. If the implementation changes later – for example, due to additional fallback mechanisms or logging – this can be done within the helper interface without having to adapt all UI components.\nI18nSupport thus forms a small but effective abstraction layer on top of the Vaadin i18n API. It ensures that translations can be used consistently throughout the application with as little boilerplate code as possible.\nDetermining the user locale # Once translation resources have been structured and convenient access has been provided via I18nSupport, the next step is to ask a central question: What language should the application use when it is first launched?\nIn web applications, this decision is usually made based on the browser\u0026rsquo;s language preferences. Modern browsers send a so-called Accept-Language header with every HTTP request that contains a prioritised list of preferred languages. Based on this information, the application can decide which locale to use initially.\nVaadin Flow already takes this information into account during user interface initialisation. When building a new UI, the browser\u0026rsquo;s locale is automatically detected and set as the current locale. This allows an application to select the appropriate language when the interface is first rendered.\nIn practice, however, it rarely makes sense to support every language reported by the browser fully. Many applications only offer a limited number of translations. Therefore, the browser-reported locale must first be compared with the application\u0026rsquo;s actually supported languages.\nIn the URL shortener project, this matching is implemented via a small helper class called LocaleSelection. This class defines the application\u0026rsquo;s supported languages and provides methods to map a requested locale to an available variant.\nA simplified example looks like this:\npublic static Locale match(Locale requested) { if (requested == null) { return EN; } String language = requested.getLanguage(); return SUPPORTED.stream() .filter(l -\u0026gt; l.getLanguage().equals(language)) .findFirst() .orElse(EN); } The method first checks which language the browser requested. It then checks whether this language is among the application\u0026rsquo;s supported languages. If this is the case, the corresponding locale is adopted. Otherwise, the application\u0026rsquo;s default language is used.\nThis strategy ensures that the application remains consistent even if a browser reports a language for which there are no complete translations. At the same time, the implementation remains clear and easily expandable.\nThe next step is how to keep the chosen language stable during a session. Therefore, in the following chapter, we will look at how a user\u0026rsquo;s locale is stored and managed within the Vaadin session.\nSession-based voice management # After describing in the previous chapter how the initial language of an application can be determined based on the browser locale, another important question arises in practice: How does this decision remain stable throughout the application\u0026rsquo;s use?\nIn a typical web application, a user session consists of multiple HTTP requests. During this time, the user expects the interface to be consistently displayed in the same language. If the language were to be determined exclusively by the browser locale for each request, this could lead to unexpected changes – especially if a user manually switches languages within the application.\nFor this reason, the URL shortener project stores the selected language within the Vaadin session. The session represents a server-side context that is retained across multiple requests and is therefore well-suited to storing user-specific settings, such as the current locale.\nVaadin itself already offers basic support for this. Each VaadinSession has an associated locale that can be set and queried by the application. If this locale is changed, it directly affects the resolution of translation keys in the user interface.\nIn the project, the session locale is also managed using the LocaleSelection helper class. In addition to the previously shown method for selecting a supported language, this class also provides methods to store or read the locale in the session.\nA simplified example for storing the locale looks like this:\npublic static void setToSession(VaadinSession session, Locale locale) { session.setAttribute(SESSION_KEY, locale); } When changing the language, the selected locale is saved in the session and set directly on the current UI. This ensures that the new language becomes active immediately and is retained for all subsequent interactions within the same session.\nThis approach has several advantages. For one, a user\u0026rsquo;s language remains consistent throughout the session. On the other hand, the language can be changed at any time by a user action without the need for additional persistence mechanisms such as cookies or database entries.\nIn addition, the implementation remains deliberately simple. The application uses only the existing Vaadin mechanisms and adds only a small helper class for central management of the supported locales.\nThe next chapter shows how this session-based locale is eventually linked to the user interface and how users can switch languages directly within the application.\nLanguage switching in the user interface # After describing in the previous chapter how to store the selected language in the Vaadin session, the practical question now arises: how can users change this language in the first place? Internationalisation is only useful if users can actively switch the interface language.\nIn the URL shortener project, language switching has been integrated directly into the MainLayout. This layout forms the central structure of the user interface and includes, among other things, the navigation bar and various status and control elements. This makes it ideal for placing a compact language switch.\nInstead of a classic drop-down menu, the application uses a small group of buttons, each representing a language. The buttons are displayed as flags, making the desired language recognisable at a glance.\nThe user interface shows three options:\nGerman English Finnish Each of these buttons changes the current locale when clicked. Technically, the first step is to check which language has been chosen. This language is then set in both the current UI and the Vaadin session.\nThe switch\u0026rsquo;s specific design is deliberately kept small and does not require any additional components or dependencies. Instead of a ComboBox, a horizontal button group is rendered, which acts like a \u0026ldquo;segmented control\u0026rdquo;: three round buttons in a slightly separated bar. The bar itself is merely a horizontal layout, with styling handled via Lumo CSS variables, so the look blends harmoniously with the rest of the theme.\nWhen the switch is created, the currently effective locale is first determined. It is crucial that not only the browser locale is considered, but that a value already stored in the session takes precedence. This is exactly what LocaleSelection.resolveAndStore(\u0026hellip;) is used for. The result is the \u0026ldquo;current\u0026rdquo; locale, which is used to mark the active button.\nprivate Component createLanguageSwitch() { Locale current = LocaleSelection.resolveAndStore( VaadinSession.getCurrent(), UI.getCurrent().getLocale() ); Button de = flagButton(LocaleSelection.DE, current); Button en = flagButton(LocaleSelection.EN, current); Button fi = flagButton(LocaleSelection.FI, current); HorizontalLayout bar = new HorizontalLayout(de, en, fi); bar.setSpacing(false); bar.getStyle() .set(\u0026#34;gap\u0026#34;, \u0026#34;6px\u0026#34;) .set(\u0026#34;padding\u0026#34;, \u0026#34;2px\u0026#34;) .set(\u0026#34;border-radius\u0026#34;, \u0026#34;999px\u0026#34;) .set(\u0026#34;background\u0026#34;, \u0026#34;var(--lumo-contrast-5pct)\u0026#34;); return bar; } The actual logic lies in creating a single button. Each button only gets a chip with an emoji flag as content. This seems trivial, but it\u0026rsquo;s practical: it requires no assets, no additional files, and no build steps. Styling and interaction take place directly on the button.\nThree points are important here: First, the buttons are shaped into a compact, round form so that they are perceived as a group that belongs together. Second, the active state is calculated based on the language (getLanguage()), because regional variants such as de-DE or de-AT should not affect the UI. Third, a click triggers the actual locale switch.\nprivate Button flagButton(Locale locale, Locale current) { String flag = flagEmoji(locale); Button b = new Button(new Span(flag)); b.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE); b.getStyle() .set(\u0026#34;width\u0026#34;, \u0026#34;38px\u0026#34;) .set(\u0026#34;height\u0026#34;, \u0026#34;32px\u0026#34;) .set(\u0026#34;min-width\u0026#34;, \u0026#34;38px\u0026#34;) .set(\u0026#34;border-radius\u0026#34;, \u0026#34;999px\u0026#34;) .set(\u0026#34;padding\u0026#34;, \u0026#34;0\u0026#34;) .set(\u0026#34;line-height\u0026#34;, \u0026#34;1\u0026#34;) .set(\u0026#34;font-size\u0026#34;, \u0026#34;18px\u0026#34;); boolean active = locale.getLanguage().equalsIgnoreCase(current.getLanguage()); applyActiveStyle(b, active); b.addClickListener(e -\u0026gt; switchLocale(locale)); Tooltip (optional) b.getElement().setProperty(\u0026#34;title\u0026#34;, locale.getLanguage().toUpperCase()); return b; } For the active state, no additional theme variant is deliberately introduced; instead, a minimal style is applied: a slight background tint and an outline ring in the primary colour. Thus, the component remains visually restrained but unambiguous.\nprivate void applyActiveStyle(Button b, boolean active) { if (active) { b.getStyle() .set(\u0026#34;background\u0026#34;, \u0026#34;var(--lumo-primary-color-10pct)\u0026#34;) .set(\u0026#34;outline\u0026#34;, \u0026#34;2px solid var(--lumo-primary-color-50pct)\u0026#34;); } else { b.getStyle() .remove(\u0026#34;background\u0026#34;) .remove(\u0026#34;outline\u0026#34;); } } The locale change itself is deliberately implemented \u0026ldquo;robustly\u0026rdquo;. First, the selected locale is matched to a supported language. It is then set in both the session and the UI. The final reload() ensures that all views, dialogues, and components resolve their texts from the resource bundles again. This avoids the additional complexity that would otherwise be caused by LocaleChangeObserver implementations across many components.\nprivate void switchLocale(Locale selected) { UI ui = UI.getCurrent(); VaadinSession session = VaadinSession.getCurrent(); Locale effective = LocaleSelection.match(selected); LocaleSelection.setToSession(session, effective); session.setLocale(effective); ui.setLocale(effective); ui.getPage().reload(); } Finally, the display of the flags is encapsulated via a small mapping method. This keeps the display in a central place and can be easily adapted later – for example, if US is to be used instead of EN or if other languages are added.\n_—\u0026lt; IMG DE\u0026gt; – this is a replacement of the emoji — ___\nprivate string flagEmoji(Locale locale) { return switch (locale.getLanguage()) { case \u0026#34;de\u0026#34; -\u0026gt; \u0026#34;\u0026lt;IMG DE\u0026gt;\u0026#34;; case \u0026#34;fi\u0026#34; -\u0026gt; \u0026#34;\u0026lt;IMG FI\u0026gt;\u0026#34;; default -\u0026gt; \u0026#34;\u0026lt;IMG EN\u0026gt;\u0026#34;; EN }; } The process consists of several steps. First, the desired language is compared with the application\u0026rsquo;s supported languages. Then the locale is set in both the session and the current UI. As a result, the new language is immediately available.\nThe final page reload ensures that all UI components are re-rendered and that all translations are consistently resolved with the new locale. This approach is deliberately kept simple and avoids additional complexity within the component logic.\nBy integrating language switching into the MainLayout, this feature is available in every view of the application. Users can switch languages at any time without leaving the current page.\nThis means that the internationalisation of the user interface is fully integrated into the application. Translation resources, locale discovery, session management, and language switching are now intertwined, enabling a multilingual interface with comparatively low implementation effort.\nConclusion and outlook # With internationalisation, the URL shortener has reached a point where good user-friendliness and clean architecture converge. The application supports multiple languages without the source code becoming confusing or requiring additional dependencies. Instead, the existing Vaadin Flow mechanisms are relied on, and only added where they make sense for the project.\nThe essential building blocks mesh neatly. Translations are stored as resource bundles in the vaadin-i18n directory so that Vaadin recognises them automatically. Clearly structured keys are used to extract texts from the interface and manage them centrally. I18nSupport reduces access to translations to a compact, anywhere-to-use method, keeping UI classes readable.\nThe locale determination follows a pragmatic principle. By default, the browser\u0026rsquo;s locale is used, but limited to the languages that are actually supported. This keeps the system predictable and avoids incomplete translation states. At the same time, the selected language is stored on a per-session basis, so that users retain a consistent interface during a session and their selection does not have to be \u0026ldquo;renegotiated\u0026rdquo; with each request.\nFinally, the whole thing becomes particularly visible in the user interface. The language switch in the MainLayout is deliberately minimalist, but on the UX side, it is much more pleasant than a nested menu. Three flag buttons are enough to quickly change the language, and the active state is immediately recognisable. The technical change to the locale is implemented robustly across the session and UI. The final reload is not a workaround, but a conscious design decision: It ensures that all views, dialogues, and components are consistently rendered in the new language without each component having to implement additional observer logic.\nThis creates a solid foundation that can be expanded if necessary. An obvious next step would be to add more languages or refine translations without changing the code. For more complex language cases – such as pluralisation, number and date formatting, or language-dependent sentence modules – ICU-based formatting could be added later. It would also be possible to save the language selection not only on a session basis, but also to link it to a user profile permanently.\nFor the current state of the URL shortener, however, the solution is deliberately kept to the essentials: It is small enough to remain maintainable, but complete enough to deliver real benefits in everyday life. This is precisely the practical value of good internationalisation – it is hardly noticeable in the code but immediately noticeable to users.\n","date":"11 March 2026","externalUrl":null,"permalink":"/posts/practical-i18n-in-vaadin-resource-bundles-locale-handling-and-ui-language-switching/","section":"Posts","summary":"Modern web applications are rarely used only by users with the same language. Even internal tools often reach international teams or are used in different countries. A multilingual user interface is therefore not a luxury feature, but an important part of the user experience.\n","title":"Practical i18n in Vaadin: Resource Bundles, Locale Handling and UI Language Switching","type":"posts"},{"content":"","date":"4 March 2026","externalUrl":null,"permalink":"/tags/css/","section":"Tags","summary":"","title":"CSS","type":"tags"},{"content":"Vaadin Flow enables the development of complete web applications exclusively in Java. Components, layouts, navigation, and even complex UI structures can be modelled on the server side without working directly with HTML or JavaScript. This approach is one of the main reasons why Vaadin is especially popular in Java-centric projects.\nHowever, as a project grows in size, a typical problem arises: styling is often done directly in Java code. Vaadin allows this conveniently via the getStyle().set(\u0026hellip;) method. This allows you to adjust spacing, colours, or layout properties quickly. In small prototypes, this procedure initially seems harmless and even practical.\nThe problem with inline styling in Vaadin Flow Separation of structure and presentation: Java vs CSS Structure belongs in Java. Appearance belongs in CSS. Advantages of this separation Replace inline styles with CSS classes Case in point Step 1: Introduce a CSS class Step 2: Using the class in Java code Another example from a view Benefits of this approach Structure of CSS Files in a Vaadin Flow Application The frontend directory Integration of styles in Vaadin Separation of layout and component styles Consistent naming of CSS classes Refactoring Real Views: Examples from the URL Shortener Example 1: MainLayout Example 2: Map layout in the AboutView Example 3: Structured layout classes Refactoring as an incremental process Best Practices for CSS in Vaadin Flow Applications Java describes structure – CSS describes representation CSS Classes Instead of Inline Styles Leveraging Lumo Design Variables Semantic Class Names Use responsive layouts consciously Inline Styles as a Deliberately Used Exception Conclusion of the refactoring In larger applications, however, this practice quickly leads to a code state that is difficult to maintain. Styling information is spread over numerous views and components. Changes to the design have to be made throughout the Java code. Reusability of styles becomes virtually impossible, and the distinction between UI logic and presentation layer becomes increasingly blurred.\nA typical example in many Vaadin projects looks something like this:\ncomponent.getStyle() .set(\u0026#34;padding\u0026#34;, \u0026#34;var(--lumo-space-m)\u0026#34;) .set(\u0026#34;border-radius\u0026#34;, \u0026#34;var(--lumo-border-radius-l)\u0026#34;); Such constructs can often be found in almost every view. Over time, a mixture of layout logic, component structure and CSS definitions emerges directly in Java code.\nIn the open-source project URL-Shortener, exactly this problem became apparent. Several views contained inline styling, which was originally intended for quick UI adjustments. However, as functionality grew, it became clear that this pattern is not sustainable in the long term.\nThe goal of the following refactoring was therefore clearly defined:\nRemove inline styling from views Introduction of clear CSS classes Clean separation between structure (Java) and presentation (CSS) Better maintainability and reusability of the UI The article describes step by step how an existing Vaadin UI can be converted from inline styling to a structured CSS architecture. The examples come directly from the real application and show typical situations that occur in many Vaadin projects.\nThis is not about making Vaadin completely CSS-driven. Vaadin remains a server-side UI framework. Rather, it is about establishing a clear division of responsibilities: Java defines the structure of the user interface, while CSS handles its visual design.\nThis seemingly small change has a significant impact on the maintainability of an application – especially if a Vaadin application is developed over the years.\nThe problem with inline styling in Vaadin Flow # At first glance, the getStyle().set(\u0026hellip;) method in Vaadin Flow seems extremely practical. With just a few lines, spacing can be adjusted, layout problems corrected, or colours changed – directly in the Java code of the respective view. Especially in early project phases or during prototype development, this approach seems quick and uncomplicated.\nIn real applications, however, a different development often emerges. As a project grows in size, inline styles spread across multiple views. What were initially only small adjustments gradually became a large number of local layout and styling corrections that are implemented directly in the Java code.\nFor example, a typical snippet from a Vaadin application might look like this:\ncomponent.getStyle().set(\u0026#34;padding\u0026#34;, \u0026#34;var(--lumo-space-m)\u0026#34;); layout.getStyle().set(\u0026#34;gap\u0026#34;, \u0026#34;var(--lumo-space-l)\u0026#34;); button.getStyle().set(\u0026#34;margin-left\u0026#34;, \u0026#34;auto\u0026#34;); Each of these lines is harmless in itself. Taken together, however, they lead to several structural problems.\nFirstly, the styling is spread over the entire application. Instead of a central place where the visual appearance is defined, design decisions are spread across many different Java classes. If, for example, the spacing of a layout or the display of certain components is to change later, developers must first find the appropriate places in the code.\nSecondly, reusability is made more difficult. Many layout patterns – such as map layouts, typical spacing, or alignments – appear multiple times in an application. However, if these are defined as inline styles, they exist again in each view. Changes then have to be made in several places.\nThird, inline styling blurs the architectural separation between structure and representation. Java classes actually define the structure of the user interface, orchestrate components, and respond to user interactions. When CSS properties are integrated directly into these classes, presentation logic mixes with the UI’s structural definition.\nThis problem is particularly evident in larger Vaadin applications developed over a longer period. Small visual adjustments accumulate, and individual getStyle().set(\u0026hellip;) calls, an implicit styling strategy gradually emerges – spread across numerous classes.\nIn the open-source project URL-Shortener, exactly this pattern emerged. In several views, including MainLayout, AboutView and CreateView, layout and styling adjustments were made directly in Java code. Although the application remained functionally correct, the UI\u0026rsquo;s visual behaviour became increasingly difficult to understand.\nThe refactoring was therefore not just aimed at removing individual style calls. Instead, a clear rule should be introduced:\nJava defines the structure of the user interface – CSS defines its visual design.\nAt first, this separation seems like a small organisational change. In practice, however, it significantly improves the application\u0026rsquo;s maintainability. Styles can be defined centrally, reused, and used consistently across multiple views.\nThe following sections show step by step how this separation can be implemented in an existing Vaadin Flow application.\nSeparation of structure and presentation: Java vs CSS # Now that the problems of inline styling in Vaadin Flow have become clear, the central question arises: How can a clean separation be created between the structure of the user interface and its visual representation?\nIn classic web applications, this division is clearly defined. HTML describes the structure of the page, CSS controls the visual appearance, and JavaScript handles interactions and logic. In Vaadin Flow, this architecture shifts slightly because the UI structure is defined entirely in Java. Nevertheless, the basic idea remains: structure and presentation should be kept separate.\nStructure belongs in Java. # Java defines the structure of the user interface in a Vaadin application. These include, in particular:\nthe composition of components\nLayout Structures Navigation and routing Event Handling Data Binding A typical example is the structure of a layout:\nVerticalLayout container = new VerticalLayout(); container.add(new H2(\u0026#34;Create new short links\u0026#34;), form, actions); Here, the code describes only the structure of the interface: which components are present and how they are arranged.\nAppearance belongs in CSS. # The visual representation of the components, on the other hand, should be defined via CSS. These include, but are not limited to:\nSpacing and padding Colours and backgrounds Shadows and Borders Typography responsive layout adjustments For example, instead of setting padding directly in Java code\ncomponent.getStyle().set(\u0026ldquo;padding\u0026rdquo;, \u0026ldquo;var(\u0026ndash;lumo-space-m)\u0026rdquo;);\nA CSS class is used:\ncomponent.addClassName(\u0026ldquo;card\u0026rdquo;);\nThe actual visual definition is then in the stylesheet:\n.card { padding: var(--lumo-space-m); border-radius: var(--lumo-border-radius-l); } This division ensures that layout decisions can be managed centrally rather than spread across several views.\nAdvantages of this separation # The consistent separation between Java structure and CSS presentation has several advantages.\nFirst, the Java code will be much more readable. Views focus on the composition of the UI components without being overloaded by styling details.\nSecond, styles can be centrally managed and reused. Once a CSS class has been defined, it can be used across different views without layout rules repeating in the code.\nThird, the application\u0026rsquo;s maintainability is improved. Changes to the design do not require adjustments in numerous Java classes, but can be made centrally in the stylesheet.\nEspecially in larger Vaadin projects with several developers, it quickly becomes apparent how important this clear division of responsibility is.\nIn the next section, we will use concrete examples from the application to show how existing inline styles can be replaced by CSS classes step by step.\nReplace inline styles with CSS classes # Once the basic separation between structure (Java) and presentation (CSS) has been defined, a practical question arises in existing Vaadin applications: How can an existing codebase be gradually converted from inline styling to CSS classes?\nIn practice, the changeover usually takes place incrementally. Instead of refactoring the entire application at once, individual views or components are cleaned up first. Direct style definitions in the Java code are removed and replaced by CSS classes.\nCase in point # A common pattern in Vaadin views is to control layout behaviour directly via inline styles. An example from the application is moving a component to the right end of a layout.\nFor example, before refactoring, the code looked like this:\nstoreIndicator.getStyle().set(\u0026ldquo;margin-left\u0026rdquo;, \u0026ldquo;auto\u0026rdquo;);\nThe code serves its purpose: the component is moved to the right. The problem, however, is that this layout rule is now firmly anchored in Java code.\nStep 1: Introduce a CSS class # Instead of setting the Style property directly, a suitable CSS class is first defined. This is typically located in the application\u0026rsquo;s global stylesheet file.\n.mainlayout-right { margin-left: auto; } The advantage of this solution is that the layout rule is now centrally defined and can be reused by several components.\nStep 2: Using the class in Java code # The next step is to remove the inline style in the Java code and replace it with the CSS class.\nstoreIndicator.addClassName(\u0026ldquo;mainlayout-right\u0026rdquo;);\nThis means that the Java code remains solely responsible for the UI\u0026rsquo;s structure. The component\u0026rsquo;s visual alignment is controlled entirely by CSS.\nAnother example from a view # Larger layout definitions can also be cleaned up in this way. In many Vaadin views, for example, constructs such as:\ncontainer.getStyle() .set(\u0026#34;border-radius\u0026#34;, \u0026#34;var(--lumo-border-radius-l)\u0026#34;) .set(\u0026#34;gap\u0026#34;, \u0026#34;var(--lumo-space-l)\u0026#34;); These style definitions can be converted into a CSS class:\n.card-container { border-radius: var(--lumo-border-radius-l); gap: var(--lumo-space-l); } In Java code, the definition is then reduced to:\ncontainer.addClassName(\u0026ldquo;card-container\u0026rdquo;);\nBenefits of this approach # This refactoring results in several improvements.\nFirst , the Java code will be much more compact and easier to read. Layout details disappear from the View classes.\nSecond , styles can be managed centrally. Layout changes only need to be made at one point in the stylesheet.\nThird , it creates a consistent visual language within the application by leveraging reusable CSS classes.\nThe migration of inline styles to CSS classes is therefore a central step in keeping Vaadin applications maintainable in the long term.\nThe next chapter shows where CSS files are stored in a Vaadin Flow application and how they are correctly integrated.\nStructure of CSS Files in a Vaadin Flow Application # Now that inline styles have been removed from the Java code and replaced with CSS classes, the next important question arises: Where should these CSS definitions be stored in a Vaadin flow application?\nVaadin Flow is internally based on Web Components and uses the application\u0026rsquo;s frontend directory to provide static resources such as stylesheets, JavaScript, or images. For CSS, this results in a clear structure that has proven itself in practice.\nThe frontend directory # In a typical Vaadin application, there is a directory called frontend. This contains all resources that are loaded directly by the browser.\nFor example, a commonly used structure looks like this:\nfrontend/\n** styles/**\n** main.css**\n** layout.css**\n** components.css**\nWithin this folder, the styles can be logically organised by area of responsibility.\nmain.css contains basic styles of the application layout.css defines layout rules components.css includes reusable component styles This splitting prevents all styles from ending up in a single file and facilitates long-term maintenance.\nIntegration of styles in Vaadin # For the CSS files to take effect in the application, they must be loaded by Vaadin. This is typically done via annotations such as @CssImport.\nAn example in a central layout class might look like this:\n@CssImport(\u0026#34;./styles/main.css\u0026#34;) public class MainLayout extends AppLayout { ... } Once this annotation is in place, the styles defined in it become available throughout the application.\nAlternatively, you can use a global theme, in which styles are loaded automatically. In this case, the CSS files are located in the directory, for example:\nfrontend/themes/ /\nThis approach is particularly recommended in larger applications, as the theme serves as a central point for all design decisions.\nSeparation of layout and component styles # Another best practice is to distinguish between layout styles and component styles.\nLayout styles describe, for example:\nContainer spacing Grid or Flex Layouts responsive behavior Component styles, on the other hand, define the visual appearance of reusable UI elements such as cards, badges, or toolbars.\nThis separation prevents layout rules and visual component logic from being mixed.\nConsistent naming of CSS classes # To ensure that CSS classes remain understandable in the long term, a consistent naming convention should be used.\nOne possible pattern is based on the respective view or component:\n.about-card\n.about-hero\n.mainlayout-right\n.searchbar-container\nThis makes it immediately recognisable which area of the application a certain style belongs to.\nA clearly structured CSS organisation is the basis for ensuring that the previously introduced separation between Java and CSS is permanent.\nWhile Java continues to define the structure and behaviour of the user interface, the application\u0026rsquo;s visual appearance is centrally controlled via stylesheets. This reduces redundancies, facilitates design changes, and ensures a consistent user interface across all views.\nThe next chapter shows how concrete refactorings in the application were implemented and which patterns proved particularly helpful.\nRefactoring Real Views: Examples from the URL Shortener # The principles described so far – the separation of structure and presentation, as well as the replacement of inline styles with CSS classes – can best be understood through concrete examples. In the open-source project URL-Shortener, several views were gradually refactored to implement this separation.\nThis chapter shows how existing Vaadin code has been adapted and which patterns have proven to be particularly practical.\nExample 1: MainLayout # In the MainLayout, an inline style was originally used to move an element to the right within a layout.\nBefore refactoring, the code looked like this:\nstoreIndicator.getStyle().set(\u0026ldquo;margin-left\u0026rdquo;, \u0026ldquo;auto\u0026rdquo;);\nThis solution works technically flawlessly, but leads to layout rules being defined directly in Java code.\nAfter the refactoring, a CSS class was introduced instead.\nJava:\nstoreIndicator.addClassName(\u0026ldquo;layout-spacer-right\u0026rdquo;);\nCSS:\n.layout-spacer-right { margin-left: auto; } The layout rule is now exclusively in the stylesheet, while the Java code only contains the structural information that this component has a specific layout role.\nExample 2: Map layout in the AboutView # The AboutView uses multiple containers that are visually represented as maps. Originally, these layout rules were defined directly via style calls.\nTypical inline code looked something like this:\ncard.getStyle() .set(\u0026#34;background\u0026#34;, \u0026#34;var(--lumo-base-color)\u0026#34;) .set(\u0026#34;border-radius\u0026#34;, \u0026#34;var(--lumo-border-radius-l)\u0026#34;) .set(\u0026#34;box-shadow\u0026#34;, \u0026#34;var(--lumo-box-shadow-s)\u0026#34;); These styles were then converted into a reusable CSS class.\nCSS:\n.card { background: var(--lumo-base-color); border-radius: var(--lumo-border-radius-l); box-shadow: var(--lumo-box-shadow-s); padding: var(--lumo-space-l); } Java:\ncard.addClassName(\u0026ldquo;card\u0026rdquo;);\nThe advantage is that all cards in the application now have a consistent appearance, and changes can be made centrally.\nExample 3: Structured layout classes # Another pattern is to name layout roles via CSS classes explicitly. Instead of generic styles, semantic classes are introduced to describe an element\u0026rsquo;s purpose.\nExamples include:\n.searchbar-container\n.bulk-actions-bar\n.overview-grid\nSuch classes make it easier to understand the application\u0026rsquo;s CSS layout.\nRefactoring as an incremental process # It is important to note that this change need not occur in a single step. In practice, an incremental approach has proven to be effective.\nNew components are developed directly with CSS classes Existing inline styles will be refactored when the opportunity arises Recurring style patterns are gradually centralised In this way, an existing Vaadin application can be converted into a cleanly structured CSS architecture without major risks.\nThe examples from the URL shortener show that even small refactorings can have a significant impact on an application\u0026rsquo;s maintainability. Removing inline styles makes the Java code clearer, while the visual design can be maintained centrally in the stylesheet.\nThe next chapter identifies the general best practices that can be derived from this refactoring and how to apply them in future Vaadin projects.\nBest Practices for CSS in Vaadin Flow Applications # After the refactoring of the existing views is complete, the question arises: which general rules can be derived from this? In practice, it has been shown that a few simple principles are already sufficient to keep Vaadin Flow applications maintainable and consistent in the long term.\nThis chapter summarises the most important best practices that have proven themselves during the conversion of the URL Shortener project.\nJava describes structure – CSS describes representation # The most important rule is also the simplest: Java defines the structure of the user interface, CSS defines its visual design.\nIn Vaadin, the entire UI composition is modelled in Java. Layout containers, components and their hierarchy therefore clearly belong in the Java code.\nExamples of structural definitions in Java include:\nLayout containers (VerticalLayout, HorizontalLayout, FormLayout) Component Structure Event Handling Data Binding Visual aspects, on the other hand, should be regulated via CSS. These include, in particular:\nSpacing Colors Shadows Typography visual highlights This clear separation prevents UI logic and presentation details from being mixed in the same code.\nCSS Classes Instead of Inline Styles # Inline styles should only be used in exceptional cases. Instead, it is recommended to define consistent CSS classes and bind them to components via addClassName().\nExample:\ncomponent.addClassName(\u0026ldquo;card\u0026rdquo;);\nThe visual definition is then done in the stylesheet:\n.card { padding: var(--lumo-space-m); border-radius: var(--lumo-border-radius-l); } This approach ensures that layout rules can be maintained centrally.\nLeveraging Lumo Design Variables # Vaadin uses Lumo, a consistent design system that provides numerous CSS variables.\nTypical examples are:\n--lumo-space-m --lumo-border-radius-l --lumo-box-shadow-s --lumo-primary-color These variables should be used preferably, as they automatically harmonise with the chosen theme and ensure consistent spacing and colours.\nAn example:\n.card { padding: var(--lumo-space-l); border-radius: var(--lumo-border-radius-l); } Semantic Class Names # CSS classes should not only describe how something looks, but what role an element plays in the layout.\nLess helpful would be, for example, names like:\n.box1 .red-border .container-large Semantic names that are based on views or components are better:\n.about-card .searchbar-container .bulk-actions-bar .overview-grid This means that even in larger style sheets, it remains clear to which area of the application a style belongs.\nUse responsive layouts consciously # Vaadin already provides many mechanisms for responsive layouts, including FormLayout, FlexLayout, and SplitLayout.\nFor example, FormLayouts can be configured directly via Java:\nform.setResponsiveSteps( new FormLayout.ResponsiveStep(\u0026#34;0\u0026#34;, 1), new FormLayout.ResponsiveStep(\u0026#34;900px\u0026#34;, 2) ); These structural layout decisions belong in the Java code by design, as they are part of the UI composition. CSS then only complements the visual behaviour.\nInline Styles as a Deliberately Used Exception # Despite all the recommendations, there are situations in which inline styles can be useful.\nTypical examples are:\ndynamically calculated layout values short-term UI adjustments experimental prototypes However, it is important that these cases remain the exception and do not become the default style of an application.\nThe combination of a clear division of responsibility, reusable CSS classes and consistent design variables results in a much more maintainable Vaadin application.\nEspecially in projects developed over a longer period, this structure quickly pays off. Changes to the visual design can be implemented centrally, while the Java code continues to describe only the user interface\u0026rsquo;s structure and behaviour.\nConclusion of the refactoring # The switch from inline styling to a clearly structured CSS architecture may seem like a small cosmetic change at first glance. In practice, however, it quickly becomes apparent that this refactoring has a profound impact on the maintainability and comprehensibility of a Vaadin Flow application.\nIn the original state of the application, many layout and display details were anchored directly in the Java code of the views. Method calls such as getStyle().set(\u0026hellip;) provided quick UI adjustments in the short term but, in the long term, led to a mixing of structure and presentation. The result was a codebase in which design decisions were spread across numerous classes.\nWith the introduction of CSS classes and the consistent outsourcing of visual properties to style sheets, this mixing has been broken up again. Java now only describes the structure of the user interface: components, layout containers, navigation and interactions. The visual design, on the other hand, is defined centrally via CSS.\nThis separation brings several immediate benefits.\nFirst, the Java code\u0026rsquo;s readability improves significantly. Views focus on the user interface\u0026rsquo;s composition without being overwhelmed by layout details. Developers can more quickly understand which structure a view has and which components interact with each other.\nSecond, the application\u0026rsquo;s design becomes more consistent. Reusable CSS classes ensure that similar components in different views are also visually identical. Changes to the appearance can be made centrally in the stylesheet without adapting numerous Java classes.\nThirdly, this structure facilitates team collaboration. Developers can focus on the Java side of the application, while design adjustments can be made independently in CSS. Especially in larger projects, this reduces conflicts and simplifies further UI development.\nThe refactoring of the URL shortener project has shown that even relatively small changes to the style structure can have a big impact on long-term maintainability. Especially in applications developed over the years, a clear separation between structure and presentation quickly pays off.\nVaadin Flow remains a server-side UI framework that puts Java at the centre of UI development. Nevertheless, such an architecture also benefits significantly from the classic principles of web development: structure, presentation, and behaviour should be clearly separated.\nIf you apply this basic rule consistently, you will get a Vaadin application that is not only functionally stable but also remains understandable and maintainable in the long term.\n","date":"4 March 2026","externalUrl":null,"permalink":"/posts/separation-of-concerns-in-vaadin-eliminating-inline-styles/","section":"Posts","summary":"Vaadin Flow enables the development of complete web applications exclusively in Java. Components, layouts, navigation, and even complex UI structures can be modelled on the server side without working directly with HTML or JavaScript. This approach is one of the main reasons why Vaadin is especially popular in Java-centric projects.\n","title":"Separation of Concerns in Vaadin: Eliminating Inline Styles","type":"posts"},{"content":"The starting point for this article was not a strategic architecture workshop or a long-term planned migration path, but a comparatively unspectacular step: updating the version numbers in the existing project. As part of further developing my URL shortener project, a regular dependency update was due anyway. Vaadin 25 (was already available at that time, so it made sense to proceed.\nThe source code for the project can be found and is\navailable underhttps://3g3.eu/url\nThe expectations were realistic, perhaps even slightly sceptical. A new major release, new minimum requirements, a modernised stack – all of these are usually good reasons to expect at least minor adjustments or initial friction. Accordingly, there was little hope that simply checking the version numbers would be sufficient.\nWhy Vaadin 25 is more than a version leap The new minimum standard: Java 21, Jakarta EE and modern platforms Leaner stack, faster builds and less ballast. Styling \u0026amp; Theming Rethought: CSS Instead of Special Logic Before and after: styling in practice Example 1: Button styling Example 2: Layout Spacing and Consistency Example 3: Dark/Light theme without special logic Component and rendering improvements in everyday life Before and After: Rendering and Overlay Behaviour in Practice Example 1: Nested dialogues Example 2: Focus return after dialogue closure Example 3: Overlay over Grid and Scroll Container Element API, SVG and MathML: Technical UIs directly from Java Before and After: Technical Visualisation with and without Element API Example 1: Status indicator as SVG Example 2: Simple Bar Chart Example 3: Representation of a Formula with MathML However, that was exactly the case. After adapting the Vaadin version and the current Java and platform dependencies, the project could be built, started, and operated without further changes. The application behaved as before: views were rendered correctly, dialogues worked, styles were intact, and even more complex UI flows showed no abnormalities.\nThis result was remarkable in that it was not based on a super trivial demo project. The URL shortener is a mature application with a clear module structure, server-side UI logic, security mechanisms and a small number of UI components. The fact that a major upgrade was possible under these conditions without immediate follow-up work is not routine.\nIt was precisely this experience that triggered a closer look at Vaadin 25. If a framework upgrade of this magnitude works right away, the question inevitably arises: why? What decisions in the substructure make this possible? Where was stability deliberately focused? And where are the changes lurking that only become relevant on closer inspection?\nWhy Vaadin 25 is more than a version leap # Vaadin 25 does not mark a classic evolutionary step with a collection of new widgets or API extensions, but a deliberate realignment of the framework. While previous major releases often featured additional features or incremental improvements, Vaadin 25 focuses on consolidation: fewer special logic elements, less legacy, and closer alignment with established standards in modern Java and web development.\nThis realignment is a direct response to the reality of many productive enterprise applications. Today, long-running web UIs must not only be functional but also maintainable, secure, and easy to integrate for years. This is exactly where Vaadin 25 comes in. The framework views itself less as an isolated ecosystem with its own rules and more as an integral part of a contemporary Java stack.\nA central signal for this is the clear commitment to modern platforms. With Java 21 as the minimum requirement and the focus on current Jakarta and Spring versions, Vaadin is deliberately positioning itself where the rest of the enterprise Java stack is moving. Older compatibility promises are abandoned in favour of creating a clean, consistent technological underpinning.\nVaadin 25 is also a turning point conceptually. Many mechanisms that were once considered Vaadin-typical are now being questioned or simplified. Styling follows more classic CSS principles, build processes are approaching the standard Java workflow, and client-side code is becoming leaner and more transparent. The result is a framework that requires less explanation and integrates more seamlessly into existing development and operational processes.\nVaadin 25 is therefore not a release that primarily stands out for visible UI innovations. Its added value lies in its long-term perspective: more stable foundations, reduced complexity, and better integration with modern Java architectures. For developers and architects, this means one thing above all: In the future, decisions for or against Vaadin can be made more clearly along technical criteria – and less along framework specifics.\nThe new minimum standard: Java 21, Jakarta EE and modern platforms # With Vaadin 25, the framework draws a clear line and defines a new minimum standard for technology. This decision was made deliberately and is aimed at projects to be operated in the long term, using current Java platforms. Backward compatibility at all costs is abandoned in favour of clarity, consistency, and maintainability.\nThe focus is on Java 21 as a binding basis. Vaadin thus relies on the current LTS version, which brings both linguistic and runtime improvements. For Vaadin applications, this means not only access to modern language constructs but also a common denominator across UI code, backend logic, and tests. The days when UI code remained at an older language level for compatibility are over.\nClosely linked to this is the alignment with current Jakarta standards. Vaadin 25 is based on Jakarta EE 11 and Servlet 6.1, and finally says goodbye to historical javax dependencies. At first glance, this change may seem like a purely technical detail. Still, in practice, it has noticeable effects: libraries, containers, and security mechanisms now share a consistent namespace and can be integrated more cleanly. Especially in safety-relevant applications, this reduces friction losses and misconfigurations.\nThis line will also be consistently continued in the Spring environment. Vaadin 25 is geared toward Spring Boot 4 and Spring Framework 7 and does not support earlier versions. This eliminates a significant amount of compatibility code required in previous versions. At the same time, Vaadin applications are moving closer to the mainstream Spring stack, which simplifies operation, monitoring and security integration.\nThe new minimum standard also affects the entire build and toolchain area. Modern Maven and Gradle plugins, current Node versions for the frontend, and a clearer separation between development and production artefacts make Vaadin projects feel more like classic Java applications. Special profiles and project-specific workarounds are becoming less important.\nThe perspective is important: Vaadin 25 requires more discipline at the start but reduces complexity over the long term. Projects planning to migrate to Java 21 or that have already migrated to Java 21 and current platforms will benefit directly. For older applications, on the other hand, the upgrade is a deliberate strategic decision – with initial effort but a much more stable foundation for the coming years.\nLeaner stack, faster builds and less ballast. # One of the most noticeable effects of Vaadin 25 is not immediately visible in the UI; it is part of the project\u0026rsquo;s underlying architecture. The framework has been specifically streamlined in this version: transitive dependencies have been reduced, historical compatibility layers have been removed, and build processes have been simplified. The result is a stack that is much closer to a classic Java server application.\nIn previous Vaadin versions, it was not uncommon for projects to carry a significant amount of indirect dependencies. Many of these were necessary to support different platform versions, old browser strategies or alternative operating modes. With Vaadin 25, much of this ballast is eliminated. The dependencies are more clearly structured, more transparently comprehensible and overall more manageable.\nThis reduction directly affects the build process. Both Maven and Gradle builds benefit from shorter resolution times and less special configuration. In particular, the gap between development and production build has narrowed. Vaadin applications behave much more strongly than ordinary Java artefacts, making it easier for new developers to get started and simplifying CI pipelines.\nStartup times in development mode also benefit from this approach. Fewer initialisation steps, clearer separation between server and client components, and modernised front-end dependencies result in a faster feedback loop. Especially in larger projects, where frequent restarts are part of the daily routine, this productivity gain should not be underestimated.\nAnother aspect is resource consumption during operations. A leaner stack does not automatically mean minimal memory usage, but it does reduce unnecessary load. Fewer libraries mean fewer classes, fewer potential conflicts, and a smaller attack surface. For production environments – especially in safety-critical or highly regulated contexts – this is a clear advantage.\nThe key thing is the demarcation: Vaadin 25 is not a minimal framework. Rather, it is about conscious reduction. Everything that is no longer necessary will be removed or simplified. This attitude runs through the entire stack and is a direct continuation of the repositioning described in the previous chapters.\nThe leaner stack thus forms an essential basis for further innovations in Vaadin 25. Simplified styling, more stable UI mechanisms and clearer migration paths are only possible because the framework frees itself from legacy burdens that have grown over the years.\nStyling \u0026amp; Theming Rethought: CSS Instead of Special Logic # The styling model is one of the areas in which Vaadin 25 stands out most clearly from previous versions. While Vaadin has long brought its own concepts and abstractions around themes, variants, and style hooks, version 25 follows a clear course: CSS is understood as an equal, direct design mechanism – without unnecessary framework-specific paths.\nThis decision is less cosmetic than architectural. In many projects, the previous styling model required in-depth knowledge of Vaadin to make UI adjustments. At the same time, existing web know-how could only be used to a limited extent. Vaadin 25 specifically reduces this friction by aligning styling more closely with established web standards, making it easier to understand and maintain.\nA visible result of this approach is the new default theme Aura. It does not replace Lumo, but sets a different focus. Aura looks calmer, more modern, and less playful without pushing itself to the forefront. For productive applications, this means a consistent look and feel without extensive customisation. Lumo remains available and is particularly suitable for projects that already build heavily on it or deliberately pursue a different visual profile.\nIn addition to the visual basis, flexibility is crucial. Vaadin 25 makes it possible to change themes dynamically – depending on user roles, clients or environments, for example. Light and dark variants can be realised without far-reaching modifications. This capability is not new, but it can be implemented more cleanly and comprehensibly with the simplified styling model.\nThe material theme was deliberately removed. This decision underscores the new focus: Instead of maintaining several design languages in parallel, Vaadin focuses on a few well-integrated core themes. For projects that require a highly individualised interface, this is not a disadvantage. On the contrary, a stronger CSS focus allows your design systems to be implemented more consistently and more framework-agnostic.\nThe effect on team collaboration is also important. Designers and front-end developers can work much more directly with Vaadin 25 without having to familiarise themselves with Vaadin-specific styling concepts. At the same time, Java developers retain full control over the UI’s structure and behaviour.\nBefore and after: styling in practice # To make the difference tangible, it is worth taking a direct look at typical styling approaches before and after Vaadin 25. The examples are deliberately simplified but clearly demonstrate the paradigm shift.\nExample 1: Button styling # Before (classic Vaadin approach with theme variants and Java hooks):\nButton save = new Button(\u0026#34;Save\u0026#34;); save.addThemeVariants(ButtonVariant.LUMO_PRIMARY); save.getStyle().set(\u0026#34;background-color\u0026#34;, \u0026#34;var(--lumo-success-color)\u0026#34;); save.getStyle().set(\u0026#34;color\u0026#34;, \u0026#34;white\u0026#34;); The styling here is partly anchored in the Java code. Colours and visual details are controlled via Vaadin-specific variables and APIs, which blur the separation between structure and representation.\nAfter (Vaadin 25, CSS-first):\nButton save = new Button(\u0026#34;Save\u0026#34;); save.addClassName(\u0026#34;btn-save\u0026#34;); .btn-save { background-color: var(--vaadin-color-success); colour: white; } Responsibility is clearly separated: Java describes the structure and semantics, while CSS handles the visual appearance. The styling is easy to find, testable and customizable independently of the Java code.\nExample 2: Layout Spacing and Consistency # Before (inline styles in code):\nVerticalLayout layout = new VerticalLayout(); layout.getStyle().set(\u0026#34;padding\u0026#34;, \u0026#34;var(--lumo-space-m)\u0026#34;); layout.getStyle().set(\u0026#34;gap\u0026#34;, \u0026#34;var(--lumo-space-s)\u0026#34;); Such constructs are functional, but quickly lead to scattered styling knowledge in the code.\nAfter (CSS-based layout classes):\nVerticalLayout layout = new VerticalLayout(); layout.addClassName(\u0026#34;content-layout\u0026#34;); .content-layout { padding: 1rem; Gap: 0.5rem; } Spacing and layout rules are now centrally defined and can be adapted consistently throughout the project.\nExample 3: Dark/Light theme without special logic # Before (Vaadin-specific theme switch):\nUI.getCurrent().getElement().setAttribute(\u0026#34;theme\u0026#34;, \u0026#34;dark\u0026#34;); After (Vaadin 25, CSS-oriented):\nUI.getCurrent().getElement().getClassList().add(\u0026#34;theme-dark\u0026#34;); .theme-dark { --vaadin-color-background: #1e1e1e; --vaadin-color-text: #f5f5f5; } Changing the theme is now a clearly defined CSS mechanism that can be easily integrated with existing design systems or user preferences.\nThese examples illustrate the core of the change: Vaadin 25 consistently shifts styling decisions to their proper place. The code becomes clearer, the styling more flexible, and collaboration among backend, frontend, and design roles much more relaxed.\nComponent and rendering improvements in everyday life # While many changes in Vaadin 25 are structural, the improvements to components and rendering are concrete in day-to-day UI work. In particular, dialogues, overlays, menus, and other floating UI elements benefit from a revised architecture designed for consistency, predictability, and stability.\nIn previous Vaadin versions, problems with overlaps, loss of focus or unexpected Z-index behaviour were not uncommon – especially in more complex applications with nested dialogues or dynamically reloaded components. Such effects could usually be remedied, but required additional logic, workarounds or a deep understanding of internal Vaadin mechanisms.\nVaadin 25 relies on a unified overlay model here. Dialogues, context menus, tooltips and similar components now follow a common rendering strategy. As a result, they are more consistently positioned, respond more stably to size changes, and behave more predictably when interacting with one another. For developers, this means fewer surprises and less special treatment in the code.\nA particularly relevant aspect is focus management. Keyboard navigation, focus restoration after closing a dialogue, and simultaneous interaction with multiple overlays are much more robust in Vaadin 25. This not only improves the user experience but also enhances accessibility and testability.\nThe interaction with modern layouts also benefits from the new rendering logic. Overlays adapt better to viewport changes, respond cleanly to scroll containers, and retain their position even during dynamic UI updates. Especially in data-driven applications with many interactions, this reduces visual artefacts and inconsistencies.\nIt is important to note that these improvements hardly require any new APIs. Existing code often benefits from the upgrade itself. At the same time, previously necessary protective measures or auxiliary constructs become superfluous, simplifying the code and making it easier to read.\nBefore and After: Rendering and Overlay Behaviour in Practice # The differences between previous Vaadin versions and Vaadin 25 are particularly evident when examining typical UI scenarios in real-world applications.\nExample 1: Nested dialogues # Before (classic approach with potential overlay issues):\nDialog editDialog = new Dialog(); editDialog.add(new TextField(\u0026#34;Name\u0026#34;)); Dialog confirmDialog = new Dialog(); confirmDialog.add(new Span(\u0026#34;Save changes?\u0026#34;)); Button save = new Button(\u0026#34;Save\u0026#34;, e -\u0026gt; confirmDialog.open()); editDialog.add(save); editDialog.open(); In more complex layouts, the second dialogue may not be positioned correctly on top of the first, or focus and keyboard navigation may be lost.\nAfter (Vaadin 25, consistent overlay model):\nDialog editDialog = new Dialog(); editDialog.setModal(true); Dialog confirmDialog = new Dialog(); confirmDialog.setModal(true); Button save = new Button(\u0026#34;Save\u0026#34;, e -\u0026gt; confirmDialog.open()); editDialog.add(save); editDialog.open(); Unified overlay rendering reliably stacks dialogues, passes focus correctly, and restores it after closing—without the need for additional auxiliary constructs.\nExample 2: Focus return after dialogue closure # Before (manual focus correction necessary):\nButton openDialog = new Button(\u0026#34;Edit\u0026#34;); Dialog dialog = new Dialog(); dialog.addDialogCloseActionListener(e -\u0026gt; openDialog.focus()); openDialog.addClickListener(e -\u0026gt; dialog.open()); Such patterns were necessary to ensure clean keyboard navigation.\nAfter (Vaadin 25, automatic focus management):\nButton openDialog = new Button(\u0026#34;Edit\u0026#34;); Dialog dialog = new Dialog(); openDialog.addClickListener(e -\u0026gt; dialog.open()); Vaadin 25 reliably handles focus return. The code becomes shorter, more understandable and less error-prone.\nExample 3: Overlay over Grid and Scroll Container # Before (unexpected positioning for scroll containers):\nGrid\u0026lt;Item\u0026gt; grid = new Grid\u0026lt;\u0026gt;(Item.class); ContextMenu menu = new ContextMenu(grid); menu.addItem(\u0026#34;Details\u0026#34;, e -\u0026gt; showDetails()); Depending on the layout and scrolling behaviour, the context menu could appear offset or cut off.\nAfter (Vaadin 25, more stable positioning):\nGrid\u0026lt;Item\u0026gt; grid = new Grid\u0026lt;\u0026gt;(Item.class); ContextMenu menu = new ContextMenu(grid); menu.setOpenOnClick(true); menu.addItem(\u0026#34;Details\u0026#34;, e -\u0026gt; showDetails()); The new rendering model handles scroll containers and viewport changes more consistently, so overlays appear where the user expects them.\nThese examples show that Vaadin 25 solves many UI problems not through new APIs, but through a more robust internal architecture. For existing applications, this often means less code, fewer workarounds, and a UI that behaves stably even in borderline cases.\nElement API, SVG and MathML: Technical UIs directly from Java # With Vaadin 25, an area is gaining importance that has so far played only a minor role in many projects: direct work with the Element API. Native support for SVG and MathML significantly expands the scope for design – especially for technical, data-driven user interfaces where classic UI components reach their limits.\nUntil now, such requirements have often been the point at which additional JavaScript libraries or external front-end frameworks have been introduced. Diagrams, status graphics or mathematical representations could be integrated, but led to media breaks in the code and increased integration effort. Vaadin 25 noticeably reduces this need by exposing these technologies as full-fledged elements in the server-side UI model.\nThe Element API enables the creation and manipulation of structured DOM elements directly from Java. With the addition of SVG, graphical primitives such as lines, circles or paths can now also be modelled on the server side. For example, it can be used to create simple diagrams, status indicators or visual markers without leaving the Vaadin component model.\nA similar effect is observed with MathML. Technical applications that have to represent formulas or calculation results benefit from the fact that mathematical expressions can be described semantically correctly. The display is not only visually more precise, but also more accessible – for example, for screen readers or automated tests.\nThe decisive point is not so much the technical feasibility as the architectural consistency. By integrating SVG and MathML into the Element API, the Java backend retains control over UI logic, rendering, and data flow. There is no additional client-side state that needs to be synchronised or secured. Especially in safety-critical or highly regulated environments, this advantage should not be underestimated.\nOf course, this approach does not replace specialised charting libraries or complex visualisation tools. Vaadin 25 deliberately positions the Element API as a tool for targeted, controlled visualisations. When it comes to clear states, technical information or explanatory graphics, this approach is often more robust and maintainable than an external dependency.\nChapter 6 thus shows another facet of Vaadin 25\u0026rsquo;s realignment. The framework does not expand its capabilities through additional abstractions; instead, it provides direct access to established web standards. For developers, this means: more expressiveness, less integration effort and a UI that can be cleanly modelled beyond classic form and table views.\nBefore and After: Technical Visualisation with and without Element API # The advantages of the extended Element API are best understood through concrete examples. The following scenarios are typical of technical applications and illustrate the difference between traditional integration approaches and the Vaadin 25 solution.\nExample 1: Status indicator as SVG # Before (external JavaScript library or client-side snippet):\nDiv status = new Div(); status.getElement().setProperty(\u0026#34;innerHTML\u0026#34;, \u0026#34;\u0026lt;svg width=\u0026#39;20\u0026#39; height=\u0026#39;20\u0026#39;\u0026gt;\u0026lt;circle cx=\u0026#39;10\u0026#39; cy=\u0026#39;10\u0026#39; r=\u0026#39;8\u0026#39; fill=\u0026#39;green\u0026#39;/\u0026gt;\u0026lt;/svg\u0026gt;\u0026#34;); The SVG code is embedded here as a string. Structure, semantics, and type checking are lost, and changes are error-prone.\nAfter (Vaadin 25, SVG via Element API):\nElement svg = new Element(\u0026#34;svg\u0026#34;); svg.setAttribute(\u0026#34;width\u0026#34;, \u0026#34;20\u0026#34;); svg.setAttribute(\u0026#34;height\u0026#34;, \u0026#34;20\u0026#34;); Element circle = new Element(\u0026#34;circle\u0026#34;); circle.setAttribute(\u0026#34;cx\u0026#34;, \u0026#34;10\u0026#34;); circle.setAttribute(\u0026#34;cy\u0026#34;, \u0026#34;10\u0026#34;); circle.setAttribute(\u0026#34;r\u0026#34;, \u0026#34;8\u0026#34;); circle.setAttribute(\u0026#34;fill\u0026#34;, \u0026#34;green\u0026#34;); svg.appendChild(circle); getElement().appendChild(svg); The SVG is now a fully modelled DOM element. Changes can be made in a targeted manner, and the structure remains comprehensible and testable.\nExample 2: Simple Bar Chart # Before (passing data to client-side code):\nJsonObject data = Json.createObject(); data.put(\u0026#34;value\u0026#34;, 75); ui.getPage().executeJs(\u0026#34;renderChart($0)\u0026#34;, data); This creates additional client-side state that must be synchronised and secured.\nAfter (Vaadin 25, server-side generated SVG):\nint value = 75; Element bar = new Element(\u0026#34;rect\u0026#34;); bar.setAttribute(\u0026#34;x\u0026#34;, \u0026#34;0\u0026#34;); bar.setAttribute(\u0026#34;y\u0026#34;, \u0026#34;0\u0026#34;); bar.setAttribute(\u0026#34;width\u0026#34;, String.valueOf(value)); bar.setAttribute(\u0026#34;height\u0026#34;, \u0026#34;20\u0026#34;); bar.setAttribute(\u0026#34;fill\u0026#34;, \u0026#34;#4caf50\u0026#34;); Element svg = new Element(\u0026#34;svg\u0026#34;); svg.setAttribute(\u0026#34;width\u0026#34;, \u0026#34;100\u0026#34;); svg.setAttribute(\u0026#34;height\u0026#34;, \u0026#34;20\u0026#34;); svg.appendChild(bar); getElement().appendChild(svg); The visualisation follows the server state directly. There is no duplicate logic and no hidden client-side code.\nExample 3: Representation of a Formula with MathML # Before (text or rendered graphic):\nSpan formula = new Span(\u0026#34;E = mc^2\u0026#34;); The formula\u0026rsquo;s semantic meaning is lost, as are its accessibility and machine-evaluability.\nAfter (Vaadin 25, MathML):\nElement math = new Element(\u0026#34;math\u0026#34;); Element mrow = new Element(\u0026#34;mrow\u0026#34;); mrow.appendChild(new Element(\u0026#34;mi\u0026#34;).setText(\u0026#34;E\u0026#34;)); mrow.appendChild(new Element(\u0026#34;mo\u0026#34;).setText(\u0026#34;=\u0026#34;)); mrow.appendChild(new Element(\u0026#34;mi\u0026#34;).setText(\u0026#34;m\u0026#34;)); mrow.appendChild(new Element(\u0026#34;mo\u0026#34;).setText(\u0026#34;·\u0026#34;)); Element msup = new Element(\u0026#34;msup\u0026#34;); msup.appendChild(new Element(\u0026#34;mi\u0026#34;).setText(\u0026#34;c\u0026#34;)); msup.appendChild(new Element(\u0026#34;mn\u0026#34;).setText(\u0026#34;2\u0026#34;)); mrow.appendChild(msup); math.appendChild(mrow); getElement().appendChild(math); The formula is now semantically correct, accessible, and clearly structured – generated directly from Java.\nThese examples make it clear how Vaadin 25 can be used to implement technical visualisations without additional front-end dependencies. The Element API thus becomes a targeted tool for controlled, server-side presentation – exactly where classic components are no longer sufficient.\n","date":"16 February 2026","externalUrl":null,"permalink":"/posts/an-unexpectedly-hassle-free-upgrade/","section":"Posts","summary":"The starting point for this article was not a strategic architecture workshop or a long-term planned migration path, but a comparatively unspectacular step: updating the version numbers in the existing project. As part of further developing my URL shortener project, a regular dependency update was due anyway. Vaadin 25 (was already available at that time, so it made sense to proceed.\n","title":"An unexpectedly hassle-free upgrade","type":"posts"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/tags/import/","section":"Tags","summary":"","title":"Import","type":"tags"},{"content":" Why an import needs a UI at all # Import functions are often treated as purely technical details in applications. Data is read in, processed and then made available – ideally without further interaction. In practice, however, an import is rarely an invisible process. It marks a transition between existing system states, between old and new data, between trust and control. This is exactly where the need for a user interface arises.\nIn the URL shortener, the import is not designed to run in the background; it is an explicit, visible process. The UI does not take on the role of a technical decision-maker, but that of a transparent mediator. It creates a clearly demarcated space in which an import can occur without immediately modifying existing data. This decoupling alone justifies its own import interface.\nThe current state of the source code is available on\nGitHub:https://github.com/svenruppert/url-shortener or https://3g3.eu/url\nWhy an import needs a UI at all The entry point in the application The import dialog as a closed workspace Implementation of the ImportDialog in Vaadin File upload: Transport instead of interpretation Validation as UI state change Presentation of results without interpretation Action Control and Apply Logic Importing is always associated with uncertainty. Even if the data format is known, it remains unclear how incoming datasets interact with existing datasets. Are there any overlaps? Are conflicts arising? Are individual entries incomplete or invalid? These questions cannot be answered by a technical process alone, but require visibility. The UI makes this intermediate state tangible without prematurely evaluating it.\nInstead of treating the import as an atomic step, it is understood as a state in the URL shortener. The user interface displays this state, records it, and prevents it from being overlooked and inadvertently entered into a production database. This makes the import process controlled, even if the UI itself does not control the content.\nThe clear distribution of roles is important here. The user interface does not interpret data, validate content, or make decisions about its technical correctness. Their task is exclusively to make the current technical status of the import visible. This deliberate restraint is crucial, as it prevents UI and import logic from being conflated.\nSo the import doesn\u0026rsquo;t need a UI because it\u0026rsquo;s complex, but because it\u0026rsquo;s a transition. The surface acts as a boundary between file and system, between potential changes and existing persistence. It gives the import a place, a time and a clear context – and that\u0026rsquo;s exactly where its raison d\u0026rsquo;être lies.\nThe entry point in the application # The import does not start automatically in the URL shortener; it only starts via a deliberately designed entry point in the application. It is where users manage existing short URLs and view the current database. This clearly positions the import as an exceptional act rather than part of the everyday processing flow.\nEntry is done via an explicit action that opens a modal dialogue. This action only creates a new instance of the import dialogue and gives it the necessary dependencies. Further logic is deliberately not provided at this point. By opening the dialogue, the user leaves the application\u0026rsquo;s normal working context and enters a clearly demarcated area where the import is fully encapsulated.\nThe modal character of the dialogue fulfils an important function here. It indicates that the import is not a background process running in parallel, but a coherent process with its own state. As long as the dialogue is open, the rest of the UI state remains unchanged. There is no implicit update and no automatic data transfer.\nFrom the user guidance perspective, this creates a clear-cut. The import is not understood as an extension of the table view, but as a temporary workspace. This separation prevents import actions from being mixed with regular editing steps. At the same time, it creates a mental orientation: everything that happens within this dialogue belongs to the import, and nothing beyond it.\nFrom a technical standpoint, the UI assumes no further responsibility at this point. The entry point does not initiate validation or server communication; instead, it delegates the entire import process to the dialogue itself. Only when the user consciously uploads a file does the import process begin.\nButton importButton = new Button(\u0026#34;Import\u0026#34;); importButton.addClickListener(event -\u0026gt; { ImportDialog dialog = new ImportDialog(urlShortenerClient, () -\u0026gt; { refreshGrid(); }); dialog.open(); }); An excerpt from the corresponding view illustrates this principle. The button opens the import dialogue and optionally passes a callback, allowing a return to the normal UI context after a successful import.\nThe code makes it clear that the entry point merely instantiates and opens the dialogue. There is no pre-check or interaction with the server or import APIs. The callback is used only for downstream view updates and is not part of the import process itself.\nThus, the entry point defines not only where the import starts, but also how it is perceived. It is a clear turning point in the context of use and forms the basis for all subsequent steps in the import dialogue.\nThe import dialog as a closed workspace # When the import dialogue opens, the application deliberately switches to a well-defined working mode. The dialogue is not intended as a mere overlay, but as a standalone UI space with its own logic, state, and clearly defined boundaries. Everything that happens within this dialogue belongs exclusively to the import and has no effect on the existing database until the process is explicitly completed.\nThis partitioning is a central design element. While the dialogue is open, the rest of the application remains frozen in its previous state. Tables, filters or active selections are neither updated nor influenced. This creates a clear separation between ongoing work and the import process, providing both technical and mental orientation.\nThe import dialogue is implemented as a standalone Vaadin component. It is fully initialised when opened and has no implicit return channel to the calling view. This decoupling allows the dialog to manage its internal state independently. Neither when opened nor during the interaction is it assumed that import data already exists. The initial state is always neutral and empty.\nStructurally, the dialogue is divided into distinct sections, each representing a specific phase of import. However, these areas are not implemented as step-by-step wizards; instead, they coexist within a common framework. This keeps the dialogue fully visible at all times and avoids implicit transitions that could obscure the actual state.\nIt is particularly important that the dialogue does not make a technical interpretation of the import data. It simply provides UI surfaces where results are displayed as soon as the server delivers them. Whether these spaces remain empty or are filled depends solely on the current import state, not on assumptions or expectations of the UI.\nThe control elements of the dialogue also follow this principle. Actions such as validation or application of the import are not permanently available, but are tied to the internal state of the dialogue. As long as there is no corresponding import data, the dialogue remains inactive. The UI does not enforce an order or progression, but only reacts to clearly defined state changes.\nThe import dialogue thus acts as a controlled container for a potentially critical operation. It fully encapsulates the import, makes its current status visible, and prevents incomplete or unchecked data from taking effect unnoticed. This clear boundary underpins all subsequent steps in the import process and explains why the dialogue was deliberately designed as a closed workspace.\nImplementation of the ImportDialog in Vaadin # The import dialogue in the URL shortener clearly shows how little \u0026ldquo;framework magic\u0026rdquo; it takes to build a complete, state-driven UI with Vaadin Flow. The entire functionality is built from a few easy-to-understand building blocks: a dialogue container, standard components such as upload, grid, tabs, and button, and clear state variables that carry the dialogue throughout the import process. The result is an interface that treats the import as a separate workspace while keeping the source code manageable.\npublic final class ImportDialog extends dialog implements HasLogger { private final URLShortenerClient client; private final Upload upload = new Upload(); private byte[] zipBytes; private final Button btnValidate = new Button(\u0026#34;Validate\u0026#34;); private final Button btnApply = new Button(\u0026#34;Apply Import\u0026#34;); private final Button btnClose = new Button(\u0026#34;Close\u0026#34;); private final Div applyHint = new Div(); private string stagingId; Getting started is already remarkably simple. Dialog is a normal Java class that extends Dialogue. This means the UI is not a declarative construct or a template; it can be fully traced in Java code. At the same time, the dialogue deliberately remains stateful: it holds both the uploaded ZIP bytes and the stagingId generated by server-side validation.\nWith this structure, it\u0026rsquo;s already clear how Vaadin Flow works here: Components are objects, and the dialogue has them like ordinary fields. The UI is thus automatically addressable without requiring \u0026ldquo;UI IDs\u0026rdquo; or bindings. At the same time, the import state is not kept externally; it belongs precisely to the dialogue in which it is made visible.\nThe constructor then shows the core of the Vaadin mechanics. With just a few lines, the dialogue is configured as a modal, resizable container. There is no separate configuration file and no DSL, but pure Java calls. This allows the import dialogue\u0026rsquo;s visual framework to be understood directly.\nsetHeaderTitle(\u0026#34;Import\u0026#34;); setModal(true); setResizable(true); setDraggable(true); setWidth(\u0026#34;1100px\u0026#34;); setHeight(\u0026#34;750px\u0026#34;); Immediately afterwards, it becomes apparent how Vaadin Flow typically models interactions. The buttons are initially deactivated and are unlocked only via events. This initialisation is a central part of the state model. The dialogue always starts neutrally and does not force progress unless there is a ZIP file.\nbtnValidate.addThemeVariants(ButtonVariant.LUMO_PRIMARY); btnValidate.setEnabled(false); btnApply.addThemeVariants(ButtonVariant.LUMO_SUCCESS); btnApply.setEnabled(false); btnClose.addClickListener(_ -\u0026gt; close()); btnValidate.addClickListener(_ -\u0026gt; validate()); btnApply.addClickListener(_ -\u0026gt; applyImport()); The interaction of UI and file upload can also be read directly. The upload will be limited to ZIP files and will have a maximum size. The key point here is the in-memory handler: once the file is successfully uploaded, the bytes are placed in zipBytes and the next step is unlocked by enabling btnValidate. The transition from \u0026ldquo;upload available\u0026rdquo; to \u0026ldquo;validation possible\u0026rdquo; is thus a single, very understandable state change.\nupload.setAcceptedFileTypes(\u0026#34;.zip\u0026#34;, APPLICATION_ZIP); upload.setMaxFiles(1); upload.setMaxFileSize(IMPORT_MAX_ZIP_BYTES); UploadHandler inMemoryUploadHandler = UploadHandler .inMemory((metadata, bytes) -\u0026gt; { String fileName = metadata.fileName(); long contentLength = metadata.contentLength(); logger().info(\u0026#34;uploaded file: fileName: {} , contentLength {}\u0026#34;, fileName, contentLength); zipBytes = bytes; logger().info(\u0026#34;setting zipBytes..\u0026#34;); btnValidate.setEnabled(true); }); upload.setUploadHandler(inMemoryUploadHandler); The actual UI structure is then created in buildContent(). Here, too, the strength of Vaadin Flow becomes apparent: Layouts are components, and the dialogue is simply built by assembling components. Headings, upload area, preview summary, tabs, and a central content container are merged into a single VerticalLayout. The code reads like a description of the interface.\nvar root = new VerticalLayout( new H3(\u0026#34;Upload ZIP\u0026#34;), upload, new H3(\u0026#34;Preview\u0026#34;), summary, applyRow, tabs, tabContent ); root.setSizeFull(); root.setPadding(false); renderTab(tabContent); return root; For displaying results, the dialogue uses two grid instances and switches between them via tabs. The decisive factor is that the tab interaction does not trigger any backend logic; it only updates the visible projection. In Vaadin Flow, switching is a single listener that empties the container and uses the appropriate combination of paging bar and grid.\ntabs.addSelectedChangeListener(e -\u0026gt; renderTab(tabContent)); private void renderTab(VerticalLayout container) { container.removeAll(); container.setSizeFull(); if (tabs.getSelectedTab() == tabInvalid) { container.add(pagingInvalid, gridInvalid); container.expand(gridInvalid); } else { container.add(pagingConflicts, gridConflicts); container.expand(gridConflicts); } } At this point, it becomes clear why the implementation remains so compact despite several states. Vaadin Flow provides the UI building blocks, and the dialogue connects them via a few, clearly defined state variables and listeners. The actual import process remains server-side. The UI only needs to know when a state is reached and how to display it.\nThis makes the ImportDialog a good example of how Vaadin Flow reduces complexity: not by hiding logic, but by using a direct programming model in which UI components, events, and state converge into a single, easy-to-read Java codebase. Especially for a minimalist import function, a complete, robust interface can be created quickly without requiring additional framework layers.\nFile upload: Transport instead of interpretation # In the import dialogue, the actual import process does not begin with a technical decision, but with a purely technical step: uploading a ZIP file. This moment marks the transition from an empty, neutral dialogue state to a potentially processable import without already making any content evaluation.\nThe upload component is firmly anchored in the dialogue and visible from the beginning. It deliberately serves as the first point of interaction within the closed work area. However, their task is clearly limited. The upload is used exclusively to receive a file and make its content temporarily available in the UI context. Neither the structure nor the semantics of the data contained play a role at this point.\nThis attitude is directly reflected in the source code. The upload is configured to accept exactly one ZIP file, and the maximum file size must not be exceeded. These restrictions do not serve the purpose of technical validation; they serve only the technical validation of the UI process.\nupload.setAcceptedFileTypes(\u0026#34;.zip\u0026#34;, APPLICATION_ZIP); upload.setMaxFiles(1); upload.setMaxFileSize(IMPORT_MAX_ZIP_BYTES); Once a file is selected and uploaded, an in-memory upload handler processes it. The dialogue saves the complete contents of the ZIP file as a byte array in an internal field. At this point, there is no check to verify whether the file contains import data or is structured correctly. In the UI, the file is initially a binary block.\nUploadHandler inMemoryUploadHandler = UploadHandler .inMemory((metadata, bytes) -\u0026gt; { zipBytes = bytes; btnValidate.setEnabled(true); }); upload.setUploadHandler(inMemoryUploadHandler); These few lines mark a crucial UI state change. Upon successful upload completion, the validation button is enabled. That\u0026rsquo;s all that happens at this point. The upload alone does not trigger any server communication and does not create an import state. It merely signals that there is now enough information to trigger the next user step.\nErrors during uploads are also dealt with exclusively from a technical perspective. If a file is rejected, for example, due to an incorrect data type or a file size that is too large, the UI responds with a corresponding notification. However, there is still no evaluation of the file\u0026rsquo;s content.\nupload.addFileRejectedListener(event -\u0026gt; { String errorMessage = event.getErrorMessage(); Notification notification = Notification.show(errorMessage, 5000, Notification.Position.MIDDLE); notification.addThemeVariants(NotificationVariant.LUMO_ERROR); }); The upload section of the dialogue thus clarifies a central design principle: The user interface does not interpret imported data. It merely ensures that a file is received in a technically correct manner and transfers this state into a clearly recognisable next possibility for action.\nOnly by clicking the \u0026ldquo;Validate\u0026rdquo; button does the dialogue leave this purely technical state of preparation. The file upload thus serves as the necessary, but deliberately content-free, basis for all subsequent steps of the import process.\nValidation as UI state change # By clicking on the \u0026ldquo;Validate\u0026rdquo; button, the import dialogue leaves its purely preparatory state for the first time. Up to this point, only technical requirements were created: a file was uploaded and stored in memory without its content being interpreted or further processed. The validate() method now marks the clear transition from a passive UI state to a state-changing step that makes the import visible for the first time.\nFrom a user interface perspective, validation is not a technical review process, but a coordinated flow of multiple UI updates, all triggered by a single user action. The dialogue deliberately assumes no responsibility for the content. It asks the server to verify the uploaded ZIP content and processes only the technical response it receives.\nThe introduction to the method is correspondingly defensive. First, the dialogue state checks whether a ZIP file exists. If this is missing, the UI displays a brief notification and cancels the process. At this point, no exception is propagated, and no internal state is changed. The validation is thus clearly tied to a previous, explicit user action.\nif (zipBytes == null || zipBytes.length == 0) { Notification.show(\u0026#34;No ZIP uploaded.\u0026#34;, 2500, Notification.Position.TOP_CENTER); return; } Only when this prerequisite is fulfilled is the actual validation step triggered. The dialogue passes the complete contents of the ZIP file to the client and calls the dedicated server endpoint for import preview. For the UI, this call is a black box. It does not know the internal validation rules or the criteria by which entries are classified as new, conflicting, or invalid.\nString previewJson = client.importValidateRaw(zipBytes); The server\u0026rsquo;s response is received in its entirety as a JSON string. Instead of converting this into a fixed data model, the UI extracts the individual values relevant to representing the import state. This includes the generated stagingId, as well as the counts of new, conflicting, and invalid entries. These values are immediately visible in the interface and constitute the first concrete import state displayed by the dialogue.\nthis.stagingId = extractJsonString(previewJson, \u0026#34;stagingId\u0026#34;); int newItems = extractJsonInt(previewJson, \u0026#34;newItems\u0026#34;, 0); int conflicts = extractJsonInt(previewJson, \u0026#34;conflicts\u0026#34;, 0); int invalid = extractJsonInt(previewJson, \u0026#34;invalid\u0026#34;, 0); With these values set, the character of the dialogue changes noticeably. The previously empty preview area is populated, and the import receives an identity for the first time: the stagingId. Nevertheless, the UI remains strictly descriptive. It does not evaluate whether the figures are plausible or in a certain relationship to each other. It only shows the current technical status.\nAt the same time, the dialogue reads the result lists directly from the server response. Iterators are used to read JSON arrays for conflicts and invalid entries, and to convert them into UI-internal row objects. These are then assigned to the corresponding grids. Here, too, there is no interpretation. If an array is empty or non-existent, the tables remain empty – a state that the UI treats as completely valid.\nfor (String obj : new ItemsArrayIterator(r, \u0026#34;conflictItems\u0026#34;)) { Map\u0026lt;String, String\u0026gt; m = JsonUtils.parseJson(obj); conflictRows.add(ConflictRow.from(m)); } After the grids are filled, paging information is updated, and the associated UI components are synchronised. This step is also performed exclusively based on the previously determined result lists, without any additional feedback to the server.\nWith the completion of the validate() method, the import dialogue is in a new, stable state. The uploaded data is validated on the server side; the results are visible, and the dialogue can now decide which further actions to offer the user. Note that validation itself does not trigger an import. It only changes the UI state and creates transparency about what a subsequent import would do.\nThe validate() method is thus the central linchpin of the entire import dialogue. It combines the upload and presentation of results without making technical decisions itself. In this role, it becomes the core of the UI-controlled import process.\nPresentation of results without interpretation # After validation, the import dialogue is in a state where concrete results are available for the first time. However, these results are not interpreted, weighted, or further processed; they are only presented. The last chapter describes exactly this moment: the transformation of raw information delivered on the server side into visible UI structures – without the user interface drawing its own conclusions.\nCentral to this chapter is the realisation that the dialogue lacks a technical view of conflicts or invalid entries. He knows neither their significance nor their effects. Instead, it performs a purely mechanical task: it takes lists of results and renders them into prepared UI components.\nThis projection is done via two separate tables, each associated with a specific result type. Conflicts and invalid entries are deliberately not presented together; they are kept in separate contexts. The dialogue offers two tabs for this purpose, which the user can switch between. However, this change only changes the view of the data, not its content or state.\nThe technical basis of this representation consists of two grid components, which are already fully configured when the dialogue is set up. Columns, widths, and display logic are fixed before it is even known whether data will ever be displayed. The grids thus exist independently of the existence of results and represent a stable projection surface.\ngridConflicts.addColumn(ConflictRow::shortCode).setHeader(\u0026#34;shortCode\u0026#34;).setAutoWidth(true).setFlexGrow(0); gridConflicts.addColumn(ConflictRow::d iff).setHeader(\u0026#34;diff\u0026#34;).setAutoWidth(true).setFlexGrow(0); gridConflicts.addColumn(ConflictRow::existingUrl).setHeader(\u0026#34;existingUrl\u0026#34;).setAutoWidth(true); gridConflicts.addColumn(ConflictRow::incomingUrl).setHeader(\u0026#34;incomingUrl\u0026#34;).setAutoWidth(true); These grids are filled not through a traditional data model, but via direct iteration over JSON fragments in the server response. The UI reads each object from its corresponding array and converts it into a simple line representation. These row objects contain exactly the fields that should be displayed, no more and no less.\nfor (String obj : new ItemsArrayIterator(r, \u0026#34;conflictItems\u0026#34;)) { Map\u0026lt;String, String\u0026gt; m = JsonUtils.parseJson(obj); conflictRows.add(ConflictRow.from(m)); } It is noteworthy that the dialogue makes no assumptions about the number of expected entries or about which fields must be present. If an array is empty or cannot be read, the associated table remains empty. This state is not treated by the UI as an error; it is considered a valid result of validation.\nThe tab control also follows this principle of neutrality. The currently selected tab only determines which table is visible. When switching, no data is reloaded, and no states are changed. The UI only shows a different section of the same import state.\ntabs.addSelectedChangeListener(e -\u0026gt; renderTab(tabContent)); The presentation of results is supplemented by a simple paging component that operates only on results already loaded. It provides a better overview of large datasets but is fully local and does not execute any additional server queries. Here, too, no content filter is applied; paging is purely representation-oriented.\nThe interplay of grids, tabs, and paging results in a deliberately restrained user interface. It shows what\u0026rsquo;s there and doesn\u0026rsquo;t show anything that hasn\u0026rsquo;t been clearly delivered. Neither is there an attempt to compensate for missing data, nor are implicit assumptions made about the \u0026ldquo;actual\u0026rdquo; state of the import.\nThis makes it clear that the result displayed in the import dialogue is not an analysis tool, but a mirror. It reflects the state delivered by the server exactly and defers any further evaluation to subsequent steps in the import process.\nAction Control and Apply Logic # After uploading, validating, and displaying the results, one central question remains in the import dialogue: Can the import actually be used? The answer to this question is not implicit or automatic. Instead, it is completely controlled by the UI state.\nThe basis of this control is not an additional server call, but the already known import state. All information relevant to the decision is already available to the UI. The updateApplyState() method serves as a central hub that interprets the state and translates it into concrete UI activations or deactivations.\nThe starting point is a strictly defensive default state. Regardless of what happened before, the Apply button will be disabled first. The UI never assumes that an import is automatically applicable. Only when all conditions are explicitly fulfilled is this state lifted.\nbtnApply.setEnabled(false); btnApply.setText(\u0026#34;Apply Import\u0026#34;); The first hard test point is the presence of a stagingId. Without this identifier, there is no valid import context from a UI perspective. Even if the data is already displayed, the import remains inapplicable until a server-side confirmed staging state is reached. The UI does not treat this case as an error, but as an incomplete state.\nThen, the two result dimensions that have already been made visible in the previous chapters are considered: invalid entries and conflicts. Invalid entries generally block the import. As soon as at least one invalid record exists, the Apply function remains deactivated, and the dialogue explicitly communicates this state via a hint text. The UI does not force a correction; it simply makes it clear that an import is not possible under these conditions.\nConflicts, on the other hand, are treated differently. From a UI perspective, they do not represent an absolute exclusion but rather a deliberate decision-making process. The dialogue includes a checkbox for this purpose, allowing the user to specify whether conflicting entries should be skipped during import. Only with this explicit consent is the import released despite existing conflicts.\nThis differentiation is directly evident in the interplay of the checkbox, information text and apply button. Activating or deactivating the checkbox immediately triggers a reassessment of the UI state without loading or recalculating any additional data. The UI responds only to the known state.\nif (conflicts \u0026gt; 0 \u0026amp;\u0026amp; !chkSkipConflicts.getValue()) { applyHint.setText(\u0026#34;Apply disabled: \u0026#34; + conflicts + \u0026#34; conflict(s). Tick \u0026#34;Skip conflicts on apply\u0026#34; to proceed.\u0026#34;); return; } When the import is finally approved, not only does the button\u0026rsquo;s activation change, but its label does as well. This allows the UI to clearly signal the conditions under which the import will be executed. This visual notice is part of the action management and serves to ensure transparency, not to enforce professional rules.\nBy clicking \u0026ldquo;Apply Import\u0026rdquo;, the dialogue leaves its purely display and decision modes. Only at this point is another server call triggered, which actually applies the previously validated import. Up to this point, the UI has only managed states, displayed them and demanded decisions.\nThe action control thus forms the deliberate conclusion of the import dialogue. It bundles all previously built-up information and converts it into an explicit user decision. It is precisely this restraint – not to apply anything automatically, to imply anything – that makes dialogue a controlled and comprehensible tool within the application.\n","date":"9 February 2026","externalUrl":null,"permalink":"/posts/the-importance-of-ui-in-import-processes/","section":"Posts","summary":"Why an import needs a UI at all # Import functions are often treated as purely technical details in applications. Data is read in, processed and then made available – ideally without further interaction. In practice, however, an import is rarely an invisible process. It marks a transition between existing system states, between old and new data, between trust and control. This is exactly where the need for a user interface arises.\n","title":"The Importance of UI in Import Processes","type":"posts"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/tags/urlshortener/","section":"Tags","summary":"","title":"Urlshortener","type":"tags"},{"content":"Export functions are often seen as a purely technical side task: one button, one download, done. In a Vaadin-based application, however, it quickly becomes apparent that exporting is much more than writing data to a file. It is a direct extension of the UI state, an infrastructural contract between frontend and backend, and a decisive factor for maintainability and predictability.\nExport from a UI point of view: more than a download button Initial situation: functional export, but non-UI Design Goal: Export as a Deterministic UI Workflow Uniform responses as a prerequisite for clean UI logic Filter logic as a common language between the grid and export Download mechanics in Vaadin: Button ≠ Download StreamResource: Export on demand instead of in advance Paging boundaries as a protective mechanism Real JSON export from the running system Effects on maintainability and comprehensibility Conclusion This article shows how a JSON-based export was deliberately designed as a UI-driven workflow in the URL shortener project. The focus is not on file formats or complex backend abstractions, but on the clean embedding of the export in a Vaadin Flow interface: filter coupling, download mechanics, paging boundaries and clear responsibilities between UI, client and server.\n**The current source code can be found on GitHub underhttps://github.com/svenruppert/url-shortener orhttps://3g3.eu/url\nExport from a UI point of view: more than a download button # In classic web applications, export is often thought of as an isolated API endpoint. From the perspective of a Vaadin UI, this consideration falls short. For the user, export is not a technical process but a consequence of the current UI state: the filters applied, the sorting, and the paging limits.\nAn export that ignores this coupling immediately leads to cognitive breaks. The display in the grid shows a certain amount of data, but the export provides something else – be it more, less or simply different data. This is exactly where it is determined whether an application is perceived as consistent.\nThe claim in the project was therefore clear: The export is not a special function, but a mirror of the UI state.\nThis decision shapes all further design steps – from the filter generation to the download mechanics to the structure of the export data itself.\nInitial situation: functional export, but non-UI # Before the revision, an export was already technically possible. Data could be read on the server and returned to the client as JSON, ensuring the basic functionality was fulfilled. However, from a user interface perspective, this implementation introduced several structural issues that only became apparent upon closer inspection.\nThe response structures were inconsistent and required the client to interpret them in context. The meaning and interaction of HTTP status codes and response bodies were not explicitly defined; they were derived from implicit assumptions in the client code. At the same time, there was no clear connection between the export and the currently visible filters in the interface. From the UI\u0026rsquo;s perspective, it was only possible to trace the data that was actually exported indirectly. In addition, there was special logic for empty results or error cases that could not be derived consistently from the response itself but were distributed across several places in the client.\nFrom Vaadin\u0026rsquo;s perspective, the export was not an integrated UI workflow but rather an isolated technical endpoint. The UI had to provide knowledge of special cases, status codes, and response formats that were not explicitly covered by a contractually defined framework. This state of affairs was also reflected in the test landscape: tests were often based on concrete string representations or complete JSON output rather than on clearly defined functional structures. Changes to filters, sorting or response formats therefore had to be followed up in several places and carried an increased risk of unintended side effects.\nIn short, the export worked technically but did not meet the requirements for a UI-enabled, traceable, and maintainable component in a Vaadin application.\nDesign Goal: Export as a Deterministic UI Workflow # The central goal of the redesign was not \u0026ldquo;more features\u0026rdquo;, but predictability. For the Vaadin UI, this means:\nThe export uses the same filters as the grid and thus reflects the current UI state. Paging limits are deliberately set and comprehensible for developers and users alike, so that the scope and character of the export remain clearly recognisable. Success, empty results and error cases are clearly distinguishable and can be handled in the UI without special logic. At the same time, the download behaves browser-compliantly and UI-stably without affecting the current UI state.\nFrom the UI\u0026rsquo;s perspective, an export must not have its own state. He must not \u0026ldquo;think\u0026rdquo; anything, expand anything, or change anything implicitly. It is a snapshot of what the user sees – nothing more, nothing less.\nUniform responses as a prerequisite for clean UI logic # In a Vaadin application, API responses have an immediate effect on the UI code, as each response is typically translated directly into UI states, component logic, and user feedback. In contrast to purely client-side frontends, the UI logic is tightly coupled to server-side processing: Each response directly updates component state, enables or disables controls, and renders feedback to the user.\nIn this context, inconsistent response formats inevitably lead to complex if-else cascades in the UI code. Special treatments for seemingly trivial cases, such as \u0026ldquo;empty\u0026rdquo; exports or different error states, must be explicitly requested. The UI code starts by interpreting technical details of the API – such as certain HTTP status codes or the presence of individual JSON fields – instead of relying on clearly defined business signals. This not only increases the code complexity but also complicates the interface behaviour during extensions, making it harder to understand and more error-prone.\nIn the URL shortener project, this problem was solved by introducing an explicit and stable response structure. Regardless of whether an export record is empty or contains an error, the response always follows the same structure. HTTP status codes are still used to signal the rough outcome of a request, but they do not serve as the sole signifier. The actual technical information – such as the context, scope, and content of the export – is transmitted in full and consistently in JSON format.\nA simplified, real export from the system illustrates this approach:\n{ \u0026#34;formatVersion\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;mode\u0026#34;: \u0026#34;filtered\u0026#34;, \u0026#34;exportedAt\u0026#34;: \u0026#34;2026-02-05T11:28:54.582886239Z\u0026#34;, \u0026#34;total\u0026#34;: 9, \u0026#34;items\u0026#34;: [ /* subject records */ ] } This creates a clear and stable contract for the Vaadin UI. The UI code can rely on the fact that metadata such as mode, exportedAt, or total is always present and interpreted consistently. The interface no longer has to guess whether an export was successful or whether there are special cases. Instead, the process can be designed in a linear, deterministic way: metadata is evaluated, the scope is checked, and the user data is processed or reported back to the user.\nThis structure has far-reaching consequences for UI logic. Loading indicators, confirmation dialogs or error messages can be derived directly from the structured response, without additional special logic or context-dependent checks. This keeps the interface clear, predictable, and closely linked to the technical significance of the answer, rather than tied to technical special cases or implicit assumptions.\nFilter logic as a common language between the grid and export # A crucial Vaadin-specific point is the reuse of the filter logic. There is no separate export filter in the project. Instead, the export is generated exclusively from the current UI state.\nThe SearchBar acts as the only source of truth:\npublic UrlMappingListRequest buildFilter(int page, int size) { UrlMappingListRequest req = new UrlMappingListRequest(); req.setPage(page); req.setSize(size); req.setActiveState(activeState); req.setCodePart(codeField.getValue()); req.setUrlPart(urlField.getValue()); req.setFrom(from); req.setTo(to); req.setSort(sort); req.setDir(dir); return req; } This Request object is used for both grid display and export. This guarantees:\nDisplay and export thus produce identical results, since both are based on the same filter definitions. Changes to filters or collations automatically and consistently affect display and export without requiring additional code. At the same time, there are no hidden or implicit export parameters, so the export behaviour can be fully explained by the UI state.\nFrom a maintenance perspective, this is a significant advantage: if you understand the UI, you understand the export.\nDownload mechanics in Vaadin: Button ≠ Download # A common mistake in Vaadin applications is trying to start a file download directly from a button click. Technically, this is problematic: a button click primarily triggers server-side logic, whereas a download is a resource from the browser\u0026rsquo;s perspective.\nIn Vaadin, a button click is primarily a server-side UI event. The browser does not send a \u0026ldquo;classic\u0026rdquo; download request; instead, Vaadin processes the click via its UI/RPC communication (server round-trip, event listener, component update). From the browser\u0026rsquo;s perspective, this is not a normal navigation or resource retrieval. And that\u0026rsquo;s exactly why \u0026ldquo;button clicks → browser downloads file\u0026rdquo; is not reliable, because the browser typically only starts a download cleanly when it retrieves a resource (link/navigation) or submits a form – i.e. something that is perceived in the browser as a \u0026ldquo;real request for a file\u0026rdquo;.\nThe anchor (\u0026lt;a\u0026gt;) element solves this problem because it is a standard download target for the browser: it has an href attribute that points to a resource, and the download attribute signals to the browser: \u0026ldquo;This is a file\u0026rdquo;. In Vaadin, you bind this href to a StreamResource. This creates a separate HTTP request when clicking the anchor, which is not part of the Vaadin UI event flow but rather an independent resource retrieval. Only at this moment is the StreamResource \u0026ldquo;pulled\u0026rdquo;, and the export content is generated on demand.\nIn practice, this has three major advantages:\nBrowser compliance and reliability: The download is started via a mechanism that the browser natively supports. This reduces edge cases in which a download triggered by a UI event is blocked or behaves inconsistently (e.g., pop-up/download policies, timing, UI updates). Decoupling from the UI lifecycle: The download occurs in a separate request. Even if Vaadin processes UI requests in parallel, if the user clicks on or rerenders the interface, the download can continue to run stably. This is especially important if export generation takes longer or is streamed. Clean accountability: The button is purely UI/UX (icon, tooltip, permissions, enable/disable, visual feedback). The anchor is purely \u0026ldquo;transport\u0026rdquo; (browser download). The StreamResource is purely a \u0026ldquo;data supplier\u0026rdquo; (the export is generated only when needed). This separation makes the code more maintainable and reduces the side effects. Button btnExport = new Button(VaadinIcon.DOWNLOAD.create()); btnExport.setTooltipText(\u0026#34;Export current result set as ZIP\u0026#34;); btnExport.addClickListener(e -\u0026gt; exportAnchor.getElement().callJsFunction(\u0026#34;click\u0026#34;) ); The actual download behaviour is in the anchor connected to a StreamResource:\nStreamResource exportResource = new StreamResource(\u0026#34;export.zip\u0026#34;, () -\u0026gt; { UrlMappingListRequest filter = searchBar.buildFilter(1, chunkSize); return urlShortenerClient.exportAllAsZipDownload(filter); }); exportAnchor.setHref(exportResource); exportAnchor.getElement().setAttribute(\u0026#34;download\u0026#34;, true); This pattern clearly separates the responsibilities: the UI interaction is limited to the button, which serves exclusively as a trigger for export. The browser download is triggered via the anchor element and is therefore treated as a regular resource request. Finally, the data is made available via the StreamResource, which only generates the export content when it is actually downloaded.\nThe export is only generated when the browser actually retrieves the resource – not when the user clicks on it.\nStreamResource: Export on demand instead of in advance # The use of StreamResource is not a detail, but a deliberate architectural decision. The export is generated on demand while the browser reads the stream.\nThis has several advantages. On the UI side, the memory footprint remains low because the export does not need to be fully pre-generated and buffered. At the same time, the UI thread is not blocked because the data transfer occurs outside the regular UI lifecycle. The download can continue regardless of the current UI state, even if the user navigates or performs further actions during this time. If errors occur during stream generation, they can be propagated cleanly via a separate HTTP request without causing the UI state to become inconsistent.\nThe export is thus technically decoupled from the UI lifecycle, although it is logically triggered by the UI.\nPaging boundaries as a protective mechanism # Another explicitly UI-related aspect of the export implementation is the deliberate limit on export quantity. The export uses the same chunkSize as the grid in the interface and is additionally limited by a fixed upper limit. This decision ensures that the export always remains within a clearly defined framework and can be derived directly from the current UI state.\nFrom an architectural perspective, this limitation prevents the export from processing large amounts of data in an uncontrolled manner when a user triggers an export. Especially in Vaadin applications, where UI interactions are typically synchronous with server-side logic, this protective measure is crucial. It reduces the risk of heavy memory loads, long runtimes, or blocking operations that could negatively impact other users or the entire server.\nAt the same time, the paging boundary conveys a clear technical semantics to the outside world. The export is deliberately defined as an image of the currently visible result set. It mirrors exactly what the user sees in the grid, including filtering, sorting, and paging configurations. This does not imply a claim to completeness, as is typically associated with a backup.\nThis clarity is particularly relevant for user expectations. The export does not provide a complete system print or a historically complete data set, but a specifically selected excerpt. The limitation makes this character explicit and prevents misinterpretations, such as assuming that an export can fully restore the system.\nFrom a maintenance and operations perspective, the paging boundary also serves as a natural safety line. It forces us to consciously design export scenarios and, if necessary, to provide separate mechanisms for backups or mass data withdrawals. As a result, the export remains a controllable UI tool and does not insidiously become an infrastructural backdoor for unlimited data queries.\nIn summary, limiting export volume is not a technical constraint but a deliberate design decision. It combines UI state, user expectations and system stability into a consistent overall picture and underlines once again that the export in the URL shortener is understood as a UI-driven result set – and expressly not as a substitute for a backup.\nReal JSON export from the running system # The architectural decisions described above are particularly well understood from a real export of the running system. The following JSON export was generated directly from the Vaadin interface and represents a specific UI state at a defined point in time.\nEven at the top level, the export contains all the necessary contextual information to enable independent classification. The formatVersion field explicitly defines the export format version, providing a stable foundation for future extensions. Changes to the internal data model do not automatically propagate to the export contract, provided the version limit is respected.\nThe field mode is deliberately chosen to speak. The filtered value makes it unmistakably clear that this is not a complete data deduction, but a result set restricted by UI filters. This information is crucial because it prevents the export from being mistakenly interpreted as a backup. The export does not capture the entire system state; it only includes the section the user has seen in the grid.\nWith exportedAt, the exact time of snapshot creation is recorded. The export thus clearly refers to a defined system state. Later changes to individual data records are deliberately not included and can be clearly delineated on the basis of this time stamp. This context is supplemented by the total field, which indicates the number of exported data records and enables a quick plausibility check without analysing the actual user data.\nThe actual technical data is located exclusively in the items array. Each entry describes a single URL-mapping dataset, including subject-relevant properties such as shortCode, originalUrl, and active, as well as temporal attributes createdAt and, optionally, expiresAt. It is notable that these objects contain no UI or export-specific metadata. They are deliberately reduced to the technical core and could also come from other contexts in the same form.\nIt is precisely this clear separation between top-level metadata and functional user data in the items array that makes the export an explainable artefact in itself. Even without knowledge of the internal code or the Vaadin interface, it is possible to determine when the export was created, under what conditions, its scope, and where the actual technical data begins.\nThe real export thus confirms the design goals described above. It is reproducible, rich in context and clearly recognisable as a UI-driven result set. Instead of merely transporting data, it also conveys its meaning and context of creation – a property that is crucial for maintainability, analysis and long-term further processing.\nEffects on maintainability and comprehensibility # The tight coupling between the export and the UI state ensures behaviour that is predictable for developers and users alike. The export follows the same rules as the grid display and contains no hidden special paths or implicit deviations. As a result, the export automatically evolves with the UI: any adjustment to filters, sorting, or paging mechanisms has a consistent effect on both paths without requiring additional synchronisation code.\nFrom a developer\u0026rsquo;s perspective, this architecture significantly reduces cognitive load. There is no separate mental model space for exporting, as its behaviour can be completely derived from the known UI state. If you understand the grid and its filter logic, you automatically understand the export. This not only simplifies onboarding new developers but also reduces the risk of unintentional inconsistencies during refactorings or functional enhancements.\nTestability also benefits directly from this clarity. Since the export has no state and relies on stable request and response structures, it can be tested in isolation. Tests can be run with specific filter combinations and validate the resulting exports without simulating the entire UI or complex interaction sequences. At the same time, UI tests remain lean because they can focus on correctly generating the filter state.\nIn the long term, this structure improves the maintainability of the overall system. Changes to the UI do not introduce hidden side effects in the export, and conversely, further development of the export does not require parallel adjustments elsewhere. The risk of divergent logic paths between display and export is not only reduced but systematically eliminated.\nIn summary, the close integration of UI state and export logic ensures that export is not a special case in the system. It becomes a transparent, explainable, long-term, and maintainable component of the application that fits seamlessly into the existing Vaadin architecture.\nConclusion # The export in the URL shortener is not an isolated API endpoint, but an integral part of the Vaadin UI architecture. It follows the same rules as the grid, uses the same filters and respects the same boundaries.\nVaadin Flow applications in particular show that a cleanly integrated export is less a question of the file format – and much more a question of clear responsibilities, explicit contracts and a consistently conceived UI workflow.\n","date":"5 February 2026","externalUrl":null,"permalink":"/posts/20224/","section":"Posts","summary":"Export functions are often seen as a purely technical side task: one button, one download, done. In a Vaadin-based application, however, it quickly becomes apparent that exporting is much more than writing data to a file. It is a direct extension of the UI state, an infrastructural contract between frontend and backend, and a decisive factor for maintainability and predictability.\n","title":"JSON export in Vaadin Flow","type":"posts"},{"content":"","date":"22 December 2025","externalUrl":null,"permalink":"/tags/advent-2025/","section":"Tags","summary":"","title":"Advent 2025","type":"tags"},{"content":" What has happened so far # In the first part, the focus was deliberately on the user interface\u0026rsquo;s structural realignment. The previously grown, increasingly monolithic OverviewView was analysed and specifically streamlined by outsourcing key functional areas to independent UI components. With the introduction of the BulkActionsBar and the SearchBar, clearly defined building blocks were created, each assuming a specific responsibility and freeing the view from operational details.\nThis refactoring was not a cosmetic step but a conscious investment in the application\u0026rsquo;s long-term maintainability. By separating presentation, interaction, and logic, a modular foundation was created that is not only easier to test but also significantly simplifies future extensions. The OverviewView transformed from a function-overloaded central element into an orchestrating instance that brings components together instead of controlling their internal processes.\nThe second part now builds upon this foundation. While Part 1 highlighted the motivation, goals, and initial extractions, the focus now shifts consistently to the new structure of the OverviewView itself: how its role has changed, how event flows have been simplified and how the interaction of the extracted components leads to a clearer, more stable architecture.\nThe source code for this article can be found on GitHub at:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-11\nHere is a screenshot of the current development status from the user\u0026rsquo;s perspective.\nThe focus is on dividing the previously monolithic OverviewView into clearly defined, reusable components such as the BulkActionsBar and the SearchBar. This separation improves maintainability, increases clarity and makes it easier to test individual functional areas. At the same time, this will pave the way for further UI optimisations and interaction patterns to be implemented gradually in the coming days.\nNew structure of the OverviewView # After removing the subcomponents, the new OverviewView appears in a much more streamlined, clearer form. While previously a large number of elements, event handlers and logic fragments were spread across the entire class, the View is now limited to a few, clearly defined tasks: the initialisation of the core components, the setup of the grid and the orchestration of the interactions between the SearchBar, BulkActionsBar and the backend.\nThis new structure follows a simple but powerful principle: the OverviewView focuses on bringing the components together rather than on how they work internally. It defines which building blocks are displayed, how they work together, and when they need to be updated. The internal structure of the individual elements – whether it be the handling of filter changes, the management of bulk actions, or the technical logic of the DataProvider – lies entirely within the respective components.\nThis clear layout makes the OverviewView look much tidier. The previously lush sections, with interspersed event listeners, complex UI layouts, and manual error handling, have largely disappeared, either merged into outsourced components or removed. They are replaced by short, easy-to-understand methods that control the flow of the view and connect its various building blocks.\nThis reduction not only makes the OverviewView easier to maintain but also easier to expand. New functions can be integrated at appropriate points without risking damage to existing logic. At the same time, an easily testable structure is created, as the dependencies are clearly defined and the responsibilities are clearly separated.\nA central element of the new structure is the initialisation of the view, which is now clearly structured and largely free of embedded logic. This new distribution of roles can already be clearly seen in the constructor:\npublic OverviewView() { setSizeFull(); setPadding(true); setSpacing(true); add(new H2(\u0026#34;URL Shortener – Overview\u0026#34;)); initDataProvider(); var pagingBar = new HorizontalLayout(prevBtn, nextBtn, pageInfo, btnSettings); pagingBar.setDefaultVerticalComponentAlignment(CENTER); HorizontalLayout bottomBar = new HorizontalLayout(new Span(), pagingBar); bottomBar.setWidthFull(); bottomBar.expand(bottomBar.getComponentAt(0)); bottomBar.setAlignItems(CENTER); VerticalLayout container = new VerticalLayout(searchBar, bottomBar); container.setPadding(false); container.setSpacing(true); container.setWidthFull(); add(container); add(bulkBar); add(grid); configureGrid(); addListeners(); addShortCuts(); try (var _ = withRefreshGuard(false)) { searchBar.setPageSize(25); searchBar.setSortBy(\u0026#34;createdAt\u0026#34;); searchBar.setDirValue(\u0026#34;desc\u0026#34;); } catch (Exception e) { throw new RuntimeException(e); } } Here it becomes clear: The OverviewView only instantiates its components, inserts them into a layout and delegates all details to specialised classes. Neither filter logic nor bulk operations are included here – the view only orchestrates.\nAnother example of the streamlined structure is the way selection is handled in the grid. In the past, there was extensive logic for buttons, states and actions here; today, the view only controls the visibility and status of the BulkActionsBar:\ngrid.addSelectionListener(event -\u0026gt; { var all = event.getAllSelectedItems(); boolean hasSelection = !all.isEmpty(); bulkBar.setVisible(hasSelection); if (hasSelection) { int count = all.size(); String label = count == 1 ? \u0026#34;link selected\u0026#34; : \u0026#34;links selected\u0026#34;; bulkBar.selectionInfoText(count + \u0026#34; \u0026#34; + label + \u0026#34; on page \u0026#34; + currentPage); } else { bulkBar.selectionInfoText(\u0026#34;\u0026#34;); } bulkBar.setButtonsEnabled(hasSelection); }); So the view only handles showing or hiding the BulkActionsBar. The actual logic for executing the actions lies entirely in the component itself.\nThe refresh mechanic is now clearly defined as well. Instead of repeating refresh calls in many places in the code, the safeRefresh() method has been created, which centrally defines how updates are performed:\npublic void safeRefresh() { logger().info(\u0026#34;safeRefresh\u0026#34;); if (!suppressRefresh) { logger().info(\u0026#34;refresh\u0026#34;); dataProvider.refreshAll(); } } This design not only makes updating cleaner, but also prevents duplicate refreshes and unwanted infinite loops.\nThe new clarity is also reflected in the configuration of the grid, which remains complex in terms of content, but is clearly delineated and outsourced to its own methods:\nprivate void configureGrid() { logger().info(\u0026#34;configureGrid..\u0026#34;); grid.addThemeVariants(GridVariant.LUMO_ROW_STRIPES, GridVariant.LUMO_COMPACT); grid.setHeight(\u0026#34;70vh\u0026#34;); grid.setSelectionMode(Grid.SelectionMode.MULTI); configureColumShortCode(); configureColumUrl(); configureColumCreated(); configureColumActive(); configureColumExpires(); configureColumActions(); grid.addItemDoubleClickListener(ev -\u0026gt; openDetailsDialog(ev.getItem())); grid.addItemClickListener(ev -\u0026gt; { if (ev.getClickCount() == 2) openDetailsDialog(ev.getItem()); }); GridContextMenu\u0026lt;ShortUrlMapping\u0026gt; menu = new GridContextMenu\u0026lt;\u0026gt;(grid); menu.addItem(\u0026#34;Show details\u0026#34;, e -\u0026gt; e.getItem().ifPresent(this::openDetailsDialog)); menu.addItem(\u0026#34;Open URL\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().open(m.originalUrl(), \u0026#34;_blank\u0026#34;))); menu.addItem(\u0026#34;Copy shortcode\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, m.shortCode()))); menu.addItem(\u0026#34;Delete...\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; confirmDelete(m.shortCode()))); } This structure makes it clear which areas of the view have which tasks. The multi-outsourced logic ensures that each method fulfils a clearly defined function.\nSimplified event pipeline after refactoring # Refactoring has significantly reduced this complexity. The event pipeline now follows a linear, predictable flow: users interact with the UI components, which trigger clearly defined actions; the OverviewView responds with manageable control signals and finally updates the grid via a uniform refresh mechanism. Critical is the introduction of safeRefresh() and withRefreshGuard(), which prevent unnecessary or recursive updates.\nThe result is an architecture in which events are no longer propagated across the view, but run along a handful of defined interfaces in a structured manner. This is already clearly visible in the central listener setup of the OverviewView:\nprivate void addListeners() { ComponentUtil.addListener(UI.getCurrent(), MappingCreatedOrChanged.class, _ -\u0026gt; { logger().info(\u0026#34;Received MappingCreatedOrChanged -\u0026gt; refreshing overview\u0026#34;); refreshPageInfo(); safeRefresh(); }); grid.addSelectionListener(event -\u0026gt; { var all = event.getAllSelectedItems(); boolean hasSelection = !all.isEmpty(); bulkBar.setVisible(hasSelection); if (hasSelection) { int count = all.size(); String label = count == 1 ? \u0026#34;link selected\u0026#34; : \u0026#34;links selected\u0026#34;; bulkBar.selectionInfoText(count + \u0026#34; \u0026#34; + label + \u0026#34; on page \u0026#34; + currentPage); } else { bulkBar.selectionInfoText(\u0026#34;\u0026#34;); } bulkBar.setButtonsEnabled(hasSelection); }); btnSettings.addClickListener(_ -\u0026gt; new ColumnVisibilityDialog\u0026lt;\u0026gt;(grid, columnVisibilityService).open()); prevBtn.addClickListener(_ -\u0026gt; { if (currentPage \u0026gt; 1) { currentPage--; refreshPageInfo(); safeRefresh(); } }); nextBtn.addClickListener(_ -\u0026gt; { int size = Optional.ofNullable(searchBar.getPageSize()).orElse(25); int maxPage = Math.max(1, (int) Math.ceil((double) totalCount / size)); if (currentPage \u0026lt; maxPage) { currentPage++; refreshPageInfo(); safeRefresh(); } }); } This is where the new clarity becomes apparent: External events such as MappingCreatedOrChanged trigger a targeted refresh; the grid selection only affects the BulkActionsBar; the paging buttons only control page and refresh. The complex logic from earlier days is clearly divided.\nThe keyboard shortcuts also integrate seamlessly into this simplified pipeline and use the encapsulated bulk logic:\nprivate void addShortCuts() { var current = UI.getCurrent(); current.addShortcutListener(_ -\u0026gt; { if (!grid.getSelectedItems().isEmpty()) { bulkBar.confirmBulkDeleteSelected(); } }, Key.DELETE); } On the SearchBar side, event processing is also greatly relieved and follows a clear pattern. Changes to filters or settings always lead to the OverviewView and its uniform refresh mechanism:\nactiveState.addValueChangeListener(_ -\u0026gt; { holdingComponent.setCurrentPage(1); holdingComponent.safeRefresh(); }); pageSize.addValueChangeListener(e -\u0026gt; { holdingComponent.setCurrentPage(1); holdingComponent.setGridPageSize(e.getValue()); holdingComponent.safeRefresh(); }); resetBtn.addClickListener(_ -\u0026gt; { try (var _ = withRefreshGuard(true)) { resetElements(); holdingComponent.setCurrentPage(1); } catch (Exception e) { throw new RuntimeException(e); } }); The use of withRefreshGuard(true) in combination with safeRefresh() ensures that certain internal state changes do not immediately trigger a cascade refresh, but are triggered in a controlled, conscious manner.\nImprovements in the MultiAliasEditorStrict # The MultiAliasEditorStrict is a central UI element in the URL shortener, allowing you to edit and manage multiple alias variants of a shortlink. During the refactoring, this editor was also revised to better integrate with the new component architecture and, at the same time, improve the user experience. Many of the original challenges resulted from tight integration with the OverviewView and from a poorly structured internal logic that had grown over the course of development.\nOne of the most essential innovations concerns the consistency of his behaviour. The MultiAliasEditorStrict is implemented as a standalone, clearly focused UI element that is solely responsible for capturing and validating aliases:\npublic class MultiAliasEditorStrict extends VerticalLayout { private static final String RX = \u0026#34;^[A-Za-z0-9_-]{3,64}$\u0026#34;; private final Grid\u0026lt;Row\u0026gt; grid = new Grid\u0026lt;\u0026gt;(Row.class, false); private final TextArea bulk = new TextArea(\u0026#34;Aliases (comma/space/newline)\u0026#34;); private final Button insertBtn = new Button(\u0026#34;Take over\u0026#34;); private final Button validateBtn = new Button(\u0026#34;Validate all\u0026#34;); private final String baseUrl; private final Function\u0026lt;String, Boolean\u0026gt; isAliasFree; Server check (true = free) public MultiAliasEditorStrict(String baseUrl, Function\u0026lt;String, Boolean\u0026gt; isAliasFree) { this.baseUrl = baseUrl; this.isAliasFree = isAliasFree; build(); } It is already clear here that the editor has a well-defined task: it knows the preview baseUrl prefix, manages its own UI elements, and provides an isAliasFree function to check for alias conflicts on the server side.\nThe structurinterface\u0026rsquo;s interface is entirely contained within the build() method. This is where text input, toolbar and grid are assembled:\nprivate void build() { setPadding(false); setSpacing(true); bulk.setWidthFull(); bulk.setMinHeight(\u0026#34;120px\u0026#34;); bulk.setValueChangeMode(ValueChangeMode.LAZY); bulk.setClearButtonVisible(true); bulk.setPlaceholder(\u0026#34;e.g.\\nnews-2025\\npromo_x\\nabc123\u0026#34;); insertBtn.addClickListener(_ -\u0026gt; parseBulk()); validateBtn.addClickListener(_ -\u0026gt; validateAll()); var toolbar = new HorizontalLayout(insertBtn, validateBtn); toolbar.setSpacing(true); configureGrid(); add(bulk, toolbar, grid); } This makes the user interface clear: First, aliases are entered in the bulk field, then transferred to the grid via \u0026ldquo;Take over\u0026rdquo; and finally checked via \u0026ldquo;Validate all\u0026rdquo;.\nThe MultiAliasEditorStrict has also been improved in terms of structure and internal architecture. Instead of distributing logic fragments across validation, synchronisation, and UI updates in an unstructured way, these areas were clearly separated and moved to dedicated methods. This is particularly evident in the grid structure and in the validation logic.\nThe grid itself displays the alias lines, a preview and the status in a compact form:\nprivate void configureGrid() { grid.addComponentColumn(row -\u0026gt; { var tf = new TextField(); tf.setWidthFull(); tf.setMaxLength(64); tf.setPattern(RX); tf.setValue(Objects.requireNonNullElse(row.getAlias(), \u0026#34;\u0026#34;)); tf.setEnabled(row.getStatus() != Status.SAVED); tf.addValueChangeListener(ev -\u0026gt; { row.setAlias(ev.getValue()); validateRow(row); grid.getDataProvider().refreshItem(row); }); return tf; }).setHeader(\u0026#34;Alias\u0026#34;).setFlexGrow(1); grid.addColumn(r -\u0026gt; baseUrl + Objects.requireNonNullElse(r.getAlias(), \u0026#34;\u0026#34;)) .setHeader(\u0026#34;Preview\u0026#34;).setAutoWidth(true); grid.addComponentColumn(row -\u0026gt; { var lbl = switch(row.getStatus()) { case NEW -\u0026gt; \u0026#34;New\u0026#34;; case VALID -\u0026gt; \u0026#34;Valid\u0026#34;; case INVALID_FORMAT -\u0026gt; \u0026#34;Format\u0026#34;; case CONFLICT -\u0026gt; \u0026#34;Taken\u0026#34;; case ERROR -\u0026gt; \u0026#34;Error\u0026#34;; case SAVED -\u0026gt; \u0026#34;Saved\u0026#34;; }; var badge = new Span(lbl); var theme = switch (row.getStatus()) { case VALID, SAVED -\u0026gt; \u0026#34;badge success\u0026#34;; case CONFLICT, INVALID_FORMAT, ERROR -\u0026gt; \u0026#34;badge error\u0026#34;; default -\u0026gt; \u0026#34;badge\u0026#34;; }; badge.getElement().getThemeList().add(theme); if (row.getMsg() != null \u0026amp;\u0026amp; !row.getMsg().isBlank()) badge.setTitle(row.getMsg()); return badge; }).setHeader(\u0026#34;Status\u0026#34;).setAutoWidth(true); grid.addComponentColumn(row -\u0026gt; { var del = new Button(\u0026#34;✕\u0026#34;, e -\u0026gt; { var items = new ArrayList\u0026lt;\u0026gt;(grid.getListDataView().getItems().toList()); items.remove(row); grid.setItems(items); }); del.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Remove\u0026#34;); return del; }).setHeader(\u0026#34;\u0026#34;).setAutoWidth(true); grid.setItems(new ArrayList\u0026lt;\u0026gt;()); grid.setAllRowsVisible(false); grid.setHeight(\u0026#34;320px\u0026#34;); } Here, it is clearly visible how all relevant information – alias, preview URL, status, and deletion action – converges in a separate, self-contained table. The status display plays a central role in providing the user with feedback on format errors, conflicts, successful validation or entries that have already been saved.\nThe real intelligence of the editor lies in the parseBulk() and validateRow() logic. parseBulk() takes over the task of processing the free text input, detecting duplicates and creating new lines in the grid:\nprivate void parseBulk() { var text = Objects.requireNonNullElse(bulk.getValue(), \u0026#34;\u0026#34;); var tokens = Arrays.stream(text.split(\u0026#34;[,;\\\\s]+\u0026#34;)) .map(String::trim) .filter(s -\u0026gt; !s.isBlank()) .distinct() .toList(); if (tokens.isEmpty()) { Notification.show(\u0026#34;No aliases to insert\u0026#34;, 2000, Notification.Position.TOP_CENTER); return; } Set\u0026lt;String\u0026gt; existing = grid.getListDataView().getItems() .map(Row::getAlias) .filter(Objects::nonNull) .collect(Collectors.toCollection(() -\u0026gt; new TreeSet\u0026lt;\u0026gt;(String.CASE_INSENSITIVE_ORDER))); var view = grid.getListDataView(); int added = 0; for (String tok : tokens) { if (existing.contains(tok)) continue; var r = new Row(tok); validateRow(r); view.addItem(r); existing.add(tok); added++; } grid.getDataProvider().refreshAll(); bulk.clear(); Notification.show(\u0026#34;Inserted: \u0026#34; + added, 2000, Notification.Position.TOP_CENTER); } The validation of individual rows is encapsulated in validateRow() and follows a clear step-by-step model of format checking, duplicate checking, and optional server querying:\nprivate void validateRow(Row r) { var a = Objects.requireNonNullElse(r.getAlias(), \u0026#34;\u0026#34;); if (!a.matches(RX)) { r.setStatus(Status.INVALID_FORMAT); r.setMsg(\u0026#34;3–64: A–Z a–z 0–9 - _\u0026#34;); return; } long same = grid.getListDataView().getItems() .filter(x -\u0026gt; x != r \u0026amp;\u0026amp; Objects.equals(a, x.getAlias())) .count(); if (same \u0026gt; 0) { r.setStatus(Status.CONFLICT); r.setMsg(\u0026#34;Duplicate in list\u0026#34;); return; } if (isAliasFree != null) { try { if (!isAliasFree.apply(a)) { r.setStatus(Status.CONFLICT); r.setMsg(\u0026#34;Alias taken\u0026#34;); return; } } catch (Exception ex) { r.setStatus(Status.ERROR); r.setMsg(\u0026#34;Check failed\u0026#34;); return; } } r.setStatus(Status.VALID); r.setMsg(\u0026#34;\u0026#34;); } Together with the status enum and the inner row class, this results in a self-contained, easily comprehensible state machine for each alias:\npublic enum Status { NEW, VALID, INVALID_FORMAT, CONFLICT, ERROR, SAVED } public static final class Row { private string alias; private status status = Status.NEW; private String msg = \u0026#34;\u0026#34;; Row(String a) { this.alias = a; } public String getAlias() { return alias; } public void setAlias(String a) { this.alias = a; } public Status getStatus() { return status; } public void setStatus(Status s) { this.status = s; } public String getMsg() { return msg; } public void setMsg(String m) { this.msg = m; } } Overall, the revision of the MultiAliasEditorStrict shows how even smaller UI components can benefit significantly from a clean structure, clear responsibilities and consistent behaviour. The component is now more stable, more comprehensible and easier to integrate – ideal conditions for future extensions or adjustments.\nCheers Sven\n","date":"22 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-extracting-components-part-2/","section":"Posts","summary":"What has happened so far # In the first part, the focus was deliberately on the user interface’s structural realignment. The previously grown, increasingly monolithic OverviewView was analysed and specifically streamlined by outsourcing key functional areas to independent UI components. With the introduction of the BulkActionsBar and the SearchBar, clearly defined building blocks were created, each assuming a specific responsibility and freeing the view from operational details.\n","title":"Advent Calendar 2025 - Extracting Components - Part 2","type":"posts"},{"content":"","date":"22 December 2025","externalUrl":null,"permalink":"/categories/eclipsestore/","section":"Categories","summary":"","title":"EclipseStore","type":"categories"},{"content":"","date":"22 December 2025","externalUrl":null,"permalink":"/tags/eclipsestore/","section":"Tags","summary":"","title":"EclipseStore","type":"tags"},{"content":"Today marks a crucial step in the evolution of the URL shortener\u0026rsquo;s user interface. After the focus in the past few days was mainly on functional enhancements – from filter and search functions to bulk operations – this day is dedicated to a structural fine-tuning: the refactoring of central UI components. This refactoring not only serves to clean up the code but also creates a clear, modular basis for future extensions as well as a significantly improved developer experience.\nThe source code for this article can be found on GitHub at:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-11\nHere is a screenshot of the current development state from the user\u0026rsquo;s perspective.\nThe focus is on dividing the previously monolithic OverviewView into clearly defined, reusable components such as the BulkActionsBar and the SearchBar. This separation improves maintainability, increases clarity and makes it easier to test individual functional areas. At the same time, this will pave the way for further UI optimisations and interaction patterns to be implemented gradually in the coming days.\nMotivation for refactoring # As an application\u0026rsquo;s functionality increases, so does its source code complexity. Especially in a UI that is continuously expanded and adapted to new requirements, this quickly leads to monolithic structures in which presentation, logic and state management are closely interwoven. This is precisely the situation that had developed in the OverviewView over the past few days of development: a central view that had to take on more and more tasks and thus became increasingly difficult to understand, extend, and test.\nRefactoring today, therefore, starts at a fundamental point. The goal is not to introduce new features, but to create a clear, modular foundation that makes future expansions much easier. Individual UI building blocks are removed from the overloaded view and formed into independent components, with responsibility-specific tasks, clear communication channels, and significantly improved reusability. This not only reduces the complexity within the view itself, but also makes the entire architecture more robust to change.\nAt the same time, this refactoring strengthens the developer experience: The structure of the code becomes more comprehensible, components can be improved in isolation, and future changes – such as to the search logic or bulk operations – can be made in the right place in a targeted manner. The result is a UI design that is more stable, more flexible and easier to maintain in the long term.\nWhy is this step a turning point today # Today marks a critical moment in the development of the URL shortener\u0026rsquo;s user interface, as the focus shifts for the first time from new features to the code\u0026rsquo;s structural quality. While the previous steps were primarily aimed at delivering visible improvements for the user, it is now a matter of ensuring the foundation on which these functions remain solid and scalable over the long term. Such refactoring creates clarity in areas that had previously grown instead of planned – and that\u0026rsquo;s precisely what makes this day a turning point.\nBy separating the central UI building blocks, an architecture emerges that no longer consists of a single, heavyweight view but of clearly defined components that perform precisely defined tasks. This structural realignment opens up new possibilities for expansion, halts progress on technical debt reduction, and ensures that upcoming features no longer have to be integrated into a confusing block. Instead, each feature can be implemented in the right place without destabilising existing areas.\nThis step thus makes a decisive contribution to long-term development: reduced friction in the code, fewer side effects, and greater clarity. So today it\u0026rsquo;s not about visible innovations, but about the quality of the foundation – and that\u0026rsquo;s exactly what makes working on upcoming expansions much more efficient, stable and enjoyable.\nOverview of the most important changes # Today, the focus is on replacing the previously central OverviewView, which had increasingly assumed responsibility over the course of development and had become correspondingly confusing. By extracting independent components, this complexity is decomposed into clearly defined building blocks.\nAn essential part of this restructuring is the introduction of the new BulkActionsBar, which bundles all mass operations, making both the code and the user interface more straightforward. The new SearchBar also creates a dedicated area that includes search and filter functions, making it easier to expand. Both components not only simplify the view but also provide a foundation for consistent design and recurring interaction patterns throughout the user interface.\nIn addition, the internal logic of the overview has been streamlined: state management, event handling, and grid interactions have been rearranged to reduce unwanted dependencies and enable targeted future changes. Minor improvements, such as the unified formatting of specific UI components, also contribute to the overall impression of a clean, structured codebase.\nWhy are UI components removed from views? # In mature user interfaces, a close interlocking of layout, interaction logic and state management often arises. As a result, central views become increasingly extensive over time and lose their original character. A clear entry point of the application becomes a complex block whose internal logic is difficult to understand. This is exactly where the separation of independent components comes in: It serves to clearly separate responsibilities and shift complexity to where it belongs – into small, clearly defined building blocks.\nBy extracting UI components, the view is reduced to its core task: orchestrating the interaction of several well-defined elements. Each component takes on a clearly defined role – be it managing search filters, triggering bulk operations, or displaying individual UI sections. This modular approach improves readability, fosters a more natural understanding of the architecture, and significantly reduces side effects during refactoring.\nAnother advantage is reusability. Components that are only loosely coupled to the overall view can be used flexibly elsewhere, expanded, or improved in isolation. This not only creates a robust structure but also enables more sustainable development practices, allowing new functions to be implemented without major interventions in existing areas. The separation of UI components is, therefore, an essential step in keeping a growing project stable and clear in the long term.\nThe new BulkActionsBar # Bulk operations are a function that plays an increasingly important role in the daily use of a URL shortener. As the number of entries grows, the need to edit several items at the same time increases – for example, to deactivate expired links, delete outdated campaigns or manage a larger number of new entries together. Such operations are not only a convenience feature but also an essential part of efficient workflows.\nIn many projects, bulk operations are first integrated directly into the main view. This works well at first, but in the long run, it causes view overload. Buttons, health checks, and more complex interaction logics begin to blend between Grid and View. The result is a structure that is difficult to maintain and in which extensions always carry the risk of unwanted side effects.\nThe decision to outsource bulk operations to a separate component creates a clear separation: the BulkActionsBar takes over the entire UI-related part of mass actions and thus forms a dedicated functional area. It bundles all relevant actions in one place, ensures a consistent user experience, and reduces complexity in the higher-level view. This clear delineation makes it much easier to add new actions, extend existing ones, or adapt the presentation to future design requirements.\nIn this way, outsourcing bulk operations makes a decisive contribution to a more stable and flexible architecture – and ensures that the user interface remains clear and intuitive even as requirements grow.\nStructure of the BulkActionsBar component # The BulkActionsBar is an independent UI component that bundles all mass actions and presents them in a clearly structured way. Their structure follows the principle that an element should cover exactly one area of responsibility: In this case, the initiation of collection actions without itself containing logic for data processing. This keeps it easy to understand and flexible to use.\nThe central component of the BulkActionsBar is a clearly defined collection of interaction elements – typically buttons or icons – each of which triggers a specific action. These elements are compactly arranged in a horizontal layout, making them highly visible and intuitively accessible in the user interface. This is complemented by a mechanism that controls the visibility and activability of actions based on whether and how many entries are selected in the grid. In this way, the component not only remains clear but also guides the user through clear interaction instructions.\nAnother essential aspect of the structure is the abstraction of event processing. The component itself does not perform any operations; instead, it signals to the higher-level view via events or callback functions which action to trigger. This creates loose coupling between the UI and the logic, which both facilitates testing and allows adjustments to individual areas without unintentionally affecting other components.\nA practical example of this is the basic structure of the class that defines the BulkActionsBar:\npublic class BulkActionsBar extends Composite\u0026lt;HorizontalLayout\u0026gt; implements HasLogger { private final URLShortenerClient urlShortenerClient; private final Grid\u0026lt;ShortUrlMapping\u0026gt; grid; private final OverviewView holdingComponent; private final Button bulkDeleteBtn = new Button(new Icon(VaadinIcon.TRASH)); private final Button bulkSetExpiryBtn = new Button(new Icon(VaadinIcon.CLOCK)); private final Button bulkClearExpiryBtn = new Button(new Icon(VaadinIcon.CLOSE_CIRCLE)); private final Button bulkActivateBtn = new Button(new Icon(VaadinIcon.PLAY)); private final Button bulkDeactivateBtn = new Button(new Icon(VaadinIcon.STOP)); private final Span selectionInfo = new Span(); public BulkActionsBar(URLShortenerClient urlShortenerClient, Grid\u0026lt;ShortUrlMapping\u0026gt; grid, OverviewView holdingComponent) { this.urlShortenerClient = urlShortenerClient; this.grid = grid; this.holdingComponent = holdingComponent; buildBulkBar(); addListeners(); } } Here, it becomes clear how the component encapsulates all elements relevant to mass actions: It knows the URLShortenerClient, the underlying Grid\u0026lt;ShortUrlMapping\u0026gt;, and the higher-level OverviewView, yet remains clearly delimited as an independent component. All buttons and the selection information area are defined within the class and are initialised in the constructor.\nThe actual visual structure of the BulkActionsBar is encapsulated in its own method:\nprivate void buildBulkBar() { --- Common style --- bulkBar().getStyle() .set(\u0026#34;background\u0026#34;, \u0026#34;var(--lumo-contrast-5pct)\u0026#34;) .set(\u0026#34;padding\u0026#34;, \u0026#34;0.4rem 0.8rem\u0026#34;) .set(\u0026#34;border-radius\u0026#34;, \u0026#34;var(--lumo-border-radius-m)\u0026#34;) .set(\u0026#34;border-bottom\u0026#34;, \u0026#34;1px solid var(--lumo-contrast-20pct)\u0026#34;); --- button setup --- setupIconButton(bulkDeleteBtn, VaadinIcon.TRASH, \u0026#34;Delete selected links\u0026#34;, \u0026#34;var(--lumo-error-color)\u0026#34;); setupIconButton(bulkSetExpiryBtn, VaadinIcon.CALENDAR_CLOCK, \u0026#34;Set expiry for selected\u0026#34;, \u0026#34;var(--lumo-primary-color)\u0026#34;); setupIconButton(bulkClearExpiryBtn, VaadinIcon.CALENDAR_CLOCK, \u0026#34;Clear expiry for selected\u0026#34;, \u0026#34;var(--lumo-secondary-text-color)\u0026#34;); setupIconButton(bulkActivateBtn, VaadinIcon.CHECK_CIRCLE, \u0026#34;Activate selected\u0026#34;, \u0026#34;var(--lumo-success-color)\u0026#34;); setupIconButton(bulkDeactivateBtn, VaadinIcon.CLOSE_CIRCLE, \u0026#34;Deactivate selected\u0026#34;, \u0026#34;var(--lumo-error-color)\u0026#34;); bulkBar().removeAll(); selectionInfo.getStyle().set(\u0026#34;opacity\u0026#34;, \u0026#34;0.7\u0026#34;); selectionInfo.getStyle().set(\u0026#34;font-size\u0026#34;, \u0026#34;var(--lumo-font-size-s)\u0026#34;); selectionInfo.getStyle().set(\u0026#34;margin-right\u0026#34;, \u0026#34;var(--lumo-space-m)\u0026#34;); bulkBar().add( selectionInfo, bulkDeleteBtn, bulkSetExpiryBtn, bulkClearExpiryBtn, bulkActivateBtn, bulkDeactivateBtn ); bulkBar().setWidthFull(); bulkBar().setSpacing(true); bulkBar().setDefaultVerticalComponentAlignment(FlexComponent.Alignment.CENTER); bulkBar().setVisible(false); } The method shows how style, layout and interaction elements are brought together in a central place. The BulkActionsBar thus defines its own visual and functional cluster within the interface, while the URLShortenerClient and the OverviewView still handle the actual execution of the actions.\nOverall, this setup enables a clean separation of display and behaviour, provides a robust foundation for future expansion, and integrates seamlessly with the application\u0026rsquo;s modular architecture.\nUsing the BulkActionsBar in the OverviewView # The BulkActionsBar is only valuable for interaction with the OverviewView, because there it becomes clear how much outsourcing mass actions simplifies the user interface structure. While these actions were previously implemented directly in the view itself, mixing a large number of event handlers, UI elements, and logic, the BulkActionsBar now handles this entire functional area. The OverviewView only needs to define when the bar becomes visible, which entries are selected, and how to respond to the triggered actions.\nThe integration follows a clear pattern: As soon as the user selects one or more rows in the grid, the view displays the BulkActionsBar and passes it the current selection state. The component itself does not perform any operations, but signals which action has been triggered via clearly defined methods and events. This creates loose coupling, which increases clarity and significantly improves visibility.\nAnother advantage is evident when handling status changes. The OverviewView no longer has to activate or deactivate buttons individually or control complex UI dependencies. Instead, a single central feedback to the BulkActionsBar is sufficient; it updates itself. This interaction not only makes the code leaner, but also prevents errors that could easily occur with multiple scattered logics.\nA central entry point for the integration is to initialise the BulkActionsBar directly as a field of the OverviewView:\nprivate final BulkActionsBar bulkBar = new BulkActionsBar(urlShortenerClient, grid, this); This provides the view with a fully configured instance that knows the URLShortenerClient, the grid, and the view itself. The component is then integrated into the constructor:\nadd(bulkBar); add(grid); Significant is the interaction with the grid’s selection listener. Here, the view decides whether the BulkActionsBar is visible and what information it should display:\ngrid.addSelectionListener(event -\u0026gt; { var all = event.getAllSelectedItems(); boolean hasSelection = !all.isEmpty(); bulkBar.setVisible(hasSelection); if (hasSelection) { int count = all.size(); String label = count == 1 ? \u0026#34;link selected\u0026#34; : \u0026#34;links selected\u0026#34;; bulkBar.selectionInfoText(count + \u0026#34; \u0026#34; + label + \u0026#34; on page \u0026#34; + currentPage); } else { bulkBar.selectionInfoText(\u0026#34;\u0026#34;); } bulkBar.setButtonsEnabled(hasSelection); }); This section clarifies the precise distribution of roles: the view recognises the current selection state, updates the visibility of the BulkActionsBar, and forwards relevant status information. The component itself remains completely free of logic for selection management.\nThe triggering of the actual actions also remains bundled in the view. An example of this is deleting using keyboard shortcuts:\ncurrent.addShortcutListener(_ -\u0026gt; { if (!grid.getSelectedItems().isEmpty()) { bulkBar.confirmBulkDeleteSelected(); } }, Key.DELETE); This once again shows the interplay of clearly defined responsibilities: the BulkActionsBar encapsulates the action UI, and the OverviewView determines the context in which it is executed.\nOverall, the use of the BulkActionsBar in the OverviewView clearly demonstrates how a clear separation of responsibilities improves the architecture of a Vaadin application. The View refocuses on its core task – presenting and updating the data – while the component encapsulates and consistently delivers all the interaction around mass actions. The BulkActionsBar in the OverviewView demonstrates how a clear separation of responsibilities improves the architecture of a Vaadin application. The View refocuses on its core task – presenting and updating the data – while the component encapsulates and consistently delivers all the interaction around mass actions.\nCode Comparison: Before vs. After # The difference between the previous implementation of bulk operations and the new, component-based structure is immediately apparent. Whereas previously all elements, buttons and dialogues were anchored directly in the OverviewView, distributed over numerous event handlers and UI areas, the code is now much more modular. The BulkActionsBar is a standalone component that encapsulates the entire UI for bulk actions. As a result, the OverviewView not only had to accommodate less logic but also gained overall clarity.\nThe difference is evident in the reduction of responsibilities within the view. Before the refactoring, the OverviewView handled all aspects: providing the buttons, displaying the bar, handling selection, opening and executing dialogues, and providing user feedback. This resulted in an extensive, tightly coupled block of code that was difficult to maintain and error-prone.\nAfter the refactoring, the structure has improved significantly. All UI-related logic of bulk operations is now exclusively in the BulkActionsBar. The OverviewView is limited to recognising the selection and forwarding the necessary information. The component itself still triggers the actions, but the view is no longer involved in the UI’s visual or structural layout. This change improves readability, as each functional area is now located where it belongs. In addition, the risk of unintended side effects is significantly reduced, as changes to the BulkActionsBar no longer require direct intervention in the OverviewView. To clarify the difference, it is worth reviewing typical places that were previously located directly in the OverviewView and are now outsourced. A classic example is the handling of the bulk deletion operation. In the past, the OverviewView itself handled dialogue, loops, error handling, and refresh. Today, the entire logic can be found clearly closed in the BulkActionsBar:\nNew structure – outsourced bulk delete logic:\npublic void confirmBulkDeleteSelected() { var selected = grid.getSelectedItems(); if (selected.isEmpty()) { Notifications.noSelection(); return; } Dialog dialog = new Dialog(); dialog.setHeaderTitle(\u0026#34;Delete \u0026#34; + selected.size() + \u0026#34; short links?\u0026#34;); var exampleCodes = selected.stream() .map(ShortUrlMapping::shortCode) .sorted() .limit(5) .toList(); if (!exampleCodes.isEmpty()) { String preview = String.join(\u0026#34;, \u0026#34;, exampleCodes); if (selected.size() \u0026gt; 5) preview += \u0026#34;, ...\u0026#34;; dialog.add(new Text(\u0026#34;Examples: \u0026#34; + preview)); } else { dialog.add(new Text(\u0026#34;Delete selected short links?\u0026#34;)); } Button confirm = new Button(\u0026#34;Delete\u0026#34;, _ -\u0026gt; { int success = 0; int failed = 0; for (var m : selected) { try { boolean ok = urlShortenerClient.delete(m.shortCode()); if (ok) success++; else failed++; } catch (IOException ex) { logger().error(\u0026#34;Bulk delete failed for {}\u0026#34;, m.shortCode(), ex); failed++; } } dialog.close(); grid.deselectAll(); holdingComponent.safeRefresh(); Notifications.deletedAndNotDeleted(success, failed); }); confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY, LUMO_ERROR); Button cancel = new Button(\u0026#34;Cancel\u0026#34;, _ -\u0026gt; dialog.close()); dialog.getFooter().add(new HorizontalLayout(confirm, cancel)); dialog.open(); } Here you can see that the entire interaction – from the UI to the error handling to the update – now takes place within the component and no longer burdens the OverviewView.\nNew structure – View only signals the selection:\ngrid.addSelectionListener(event -\u0026gt; { var all = event.getAllSelectedItems(); boolean hasSelection = !all.isEmpty(); bulkBar.setVisible(hasSelection); if (hasSelection) { int count = all.size(); String label = count == 1 ? \u0026#34;link selected\u0026#34; : \u0026#34;links selected\u0026#34;; bulkBar.selectionInfoText(count + \u0026#34; \u0026#34; + label + \u0026#34; on page \u0026#34; + currentPage); } else { bulkBar.selectionInfoText(\u0026#34;\u0026#34;); } bulkBar.setButtonsEnabled(hasSelection); }); This shows the new role distribution: the OverviewView only controls visibility and status display. In the past, the entire operation logic would have been implemented here as well. Another example is opening the bulk set expiry dialogues. This logic is now also completely in the component and no longer distributed across the View:\nbulkSetExpiryBtn.addClickListener(_ -\u0026gt; openBulkSetExpiryDialog()); The entire dialogue logic is then located in:\nprivate void openBulkSetExpiryDialog() { ... } Result:\nThe clear separation of responsibilities results in:\nsignificantly fewer lines of code in the OverviewView, a more structured, modular architecture, less coupling, improved maintainability and expandability. Overall, this results in a more maintenance-friendly, extensible and clear structured architecture.\nSearch requirements # A practical search function is a central part of any administrative interface, especially when the number of entries is continuously growing. In the context of URL shorteners, this means that users can quickly and specifically access specific short links – regardless of whether they are looking for a particular shortcode, a URL fragment, the active status or time criteria. The previous implementation offered only simple filter options and was tightly coupled to the OverviewView, making both extensibility and maintenance difficult.\nHowever, the requirements for a modern search component go far beyond a simple text field. It must support multiple filter criteria, respond flexibly to new fields, and adapt dynamically to the backend data structure. At the same time, it should offer the user an intuitive, consistent and clutter-free interface. This includes clear input fields, understandable labels, sensible default values, and the ability to change or reset search parameters quickly.\nTechnical reliability is just as crucial as user-friendliness: the search function must not burden the backend unnecessarily, support high-performance queries, and use server-side filters efficiently. A clean separation between UI, filter logic and data retrieval not only enables a better overview, but also later extensions – such as additional filter fields, sorting options or advanced functions such as combining several criteria.\nThe introduction of the new SearchBar addresses these requirements. It serves as the comprehensive filter and control centre for the overview and ensures that the view itself is decoupled from the filter logic. In doing so, it lays the foundation for a scalable and user-friendly search and filtering experience.\nStructure and internal logic of the SearchBar # The SearchBar is designed as an independent UI component that aggregates all filter and search parameters from the OverviewView and presents them in a clearly structured format. This component aims to centralise previously scattered filter logic and significantly improve view clarity. Whereas previously individual input fields and sorting parameters were defined directly in the OverviewView, the SearchBar now assumes full responsibility for their management.\nIts structure follows a modular concept: Each input – be it a text filter, a sorting criterion, the number of displayed elements or filtering by properties such as active status or expiration date – is clearly demarcated within the component and is processed independently. This not only ensures better logical separation but also enables flexible expansion with additional filters without requiring adjustments to the view itself.\nThe internal logic of the SearchBar works closely with the backend. Based on the user-selected parameters, the component creates structured filter objects that can be passed to the server consistently. Instead of collecting individual parameters loosely or combining them in the view, they are merged in a clearly defined process: validated, normalised, and then transferred to a backend request.\nThe central method buildFilter, which creates a UrlMappingListRequest object from the UI inputs, shows what this process looks like in concrete terms:\npublic UrlMappingListRequest buildFilter(Integer page, Integer size) { UrlMappingListRequest.Builder b = UrlMappingListRequest.builder(); ActiveState activeStateValue = activeState.getValue(); logger().info(\u0026#34;buildFilter - activeState == {}\u0026#34;, activeStateValue); if (activeStateValue != null \u0026amp;\u0026amp; activeStateValue.isSet()) { b.active(activeStateValue.toBoolean()); } if (codePart.getValue() != null \u0026amp;\u0026amp; !codePart.getValue().isBlank()) { b.codePart(codePart.getValue()); } if (urlPart.getValue() != null \u0026amp;\u0026amp; !urlPart.getValue().isBlank()) { b.urlPart(urlPart.getValue()); } if (fromDate.getValue() != null \u0026amp;\u0026amp; fromTime.getValue() != null) { var zdt = ZonedDateTime.of(fromDate.getValue(), fromTime.getValue(), ZoneId.systemDefault()); b.from(zdt.toInstant()); } else if (fromDate.getValue() != null) { var zdt = fromDate.getValue().atStartOfDay(ZoneId.systemDefault()); b.from(zdt.toInstant()); } if (toDate.getValue() != null \u0026amp;\u0026amp; toTime.getValue() != null) { var zdt = ZonedDateTime.of(toDate.getValue(), toTime.getValue(), ZoneId.systemDefault()); b.to(zdt.toInstant()); } else if (toDate.getValue() != null) { var zdt = toDate.getValue().atTime(23, 59).atZone(ZoneId.systemDefault()); b.to(zdt.toInstant()); } if (sortBy.getValue() != null \u0026amp;\u0026amp; !sortBy.getValue().isBlank()) b.sort(sortBy.getValue()); if (dir.getValue() != null \u0026amp;\u0026amp; !dir.getValue().isBlank()) b.dir(dir.getValue()); if (page != null \u0026amp;\u0026amp; size != null) { b.page(page).size(size); } var filter = b.build(); logger().info(\u0026#34;buildFilter - {}\u0026#34;, filter); return filter; } The code shows how all relevant filter fields – active status, shortcode and URL parts, time windows, and sorting and paging information – are consolidated in a single place. The SearchBar thus serves as a translator between UI inputs and the domain-specific filter structure in the backend. This creates a robust interface that both reduces errors and facilitates future expansions. At the same time, the SearchBar ensures a consistent user experience. Changes to a field automatically update the results list, while sensible default values and a consistent display ensure a familiar user experience. This combination of structural clarity, technical precision, and ease of use makes the SearchBar a central building block of the project\u0026rsquo;s modern UI architecture.\nImprovements compared to the previous solution # From a technical standpoint, the new solution offers greater robustness. The processing is now handled entirely in the SearchBar, using structured data objects that are transmitted directly to the backend. This minimises errors caused by inconsistent or incomplete filter parameters. At the same time, the clear structure makes it easy to add new filters without jeopardising existing functionality.\nA key example of the improvements is how changes to the SearchBar filter are handled. Whereas previously the OverviewView had to check and react to each value, the SearchBar now handles this task independently. This is already evident in the ValueChange listeners of the individual input fields:\ncodePart.addValueChangeListener(_ -\u0026gt; holdingComponent.safeRefresh()); urlPart.addValueChangeListener(_ -\u0026gt; holdingComponent.safeRefresh()); activeState.addValueChangeListener(_ -\u0026gt; { holdingComponent.setCurrentPage(1); holdingComponent.safeRefresh(); }); pageSize.addValueChangeListener(e -\u0026gt; { holdingComponent.setCurrentPage(1); holdingComponent.setGridPageSize(e.getValue()); holdingComponent.safeRefresh(); }); These listeners make it clear that the SearchBar takes complete control of the filters\u0026rsquo; behaviour and automatically updates the view. Previously, this logic was distributed in the OverviewView. Global search has also been significantly improved. It used to be a standalone input field with no real integration, but now it dynamically controls the specific search fields and ensures consistent filters:\nglobalSearch.addValueChangeListener(e -\u0026gt; { var v = Optional.ofNullable(e.getValue()).orElse(\u0026#34;\u0026#34;); if (searchScope.getValue().equals(\u0026#34;Shortcode\u0026#34;)) { codePart.setValue(v); urlPart.clear(); } else { urlPart.setValue(v); codePart.clear(); } }); As a result, global search has become a real entry point for filter logic, rather than being an additional field with no straightforward integration.\nAnother big step forward is the introduction of the reset mechanism, which resets all filters in a targeted manner while ensuring that the UI returns to a consistent state:\nresetBtn.addClickListener(_ -\u0026gt; { try (var _ = withRefreshGuard(true)) { resetElements(); holdingComponent.setCurrentPage(1); } catch (Exception e) { throw new RuntimeException(e); } }); Finally, switching between simple and advanced search also shows the new structural quality of the SearchBar:\nadvanced.addOpenedChangeListener(ev -\u0026gt; { boolean nowClosed = !ev.isOpened(); if (nowClosed) { applyAdvancedToSimpleAndReset(); } else { setSimpleSearchEnabled(false); } }); Overall, the new SearchBar is far superior in both functionality and architecture. Their listeners, control logic, and consistent handling of filter elements form the basis for a modern, scalable, and maintenance-friendly search architecture. It is not only functionally better, but also forms the basis for a modern, scalable and maintenance-friendly search architecture.\nInteraction of the SearchBar with grid and backend # The SearchBar only shows its full strength when interacting with the grid and the backend that powers it. While it serves as a central input component for all search and filter parameters on the interface, it technically forms the link between user interaction and server-side data supply. It is precisely in this role that it becomes clear how crucial the decoupling of the filter logic from the actual view is for performance, maintainability and extensibility.\nIn the first step, the SearchBar accepts all user input – from global search texts to specific shortcode or URL parts to date ranges, sort fields and the active status. These values are no longer processed ad hoc in the OverviewView, but collected, validated and harmonised in a structured form within the SearchBar. This keeps filter states consistent and traceable, even when multiple input fields are changed simultaneously.\nThe second step is the interaction with the grid. As soon as a relevant filter value changes, the SearchBar notifies the OverviewView of the updated state. This, in turn, triggers a grid update without requiring the individual filter criteria to be known or processed. The grid then calls the backend via its DataProvider and receives a filtered subset of the data based on the SearchBar\u0026rsquo;s requirements. This creates a clearly separated yet closely interlinked system of input, data retrieval and presentation.\nOn the backend, the advantage of a structured filter is particularly evident in how the OverviewView configures its DataProvider. The SearchBar always provides a consistent filter object that is passed directly to the backend calls. Central to this is the initialisation of the DataProvider in the OverviewView:\nprivate void initDataProvider() { dataProvider = new CallbackDataProvider\u0026lt;\u0026gt;( q -\u0026gt; { final int uiSize = Optional.ofNullable(searchBar.getPageSize()).orElse(25); final int pageStart = (currentPage - 1) * uiSize; final int vLimit = q.getLimit(); final int vOffset = q.getOffset(); final int effectiveLimit = (vLimit \u0026gt; 0) ? vLimit: uiSize; final int effectiveOffset = pageStart + vOffset; final int page = (effectiveLimit \u0026gt; 0) ? (effectiveOffset / effectiveLimit) + 1 : 1; final int size = (effectiveLimit \u0026gt; 0) ? effectiveLimit: uiSize; final UrlMappingListRequest req = searchBar.buildFilter(page, size); try { final List\u0026lt;ShortUrlMapping\u0026gt; items = urlShortenerClient.list(req); return items.stream(); } catch (IOException ex) { logger().error(\u0026#34;Error fetching (page={}, size={})\u0026#34;, page, size, ex); Notifications.loadingFailed(); return Stream.empty(); } }, _ -\u0026gt; { try { final UrlMappingListRequest base = searchBar.buildFilter(null, null); totalCount = urlShortenerClient.listCount(base); final int uiSize = Optional.ofNullable(searchBar.getPageSize()).orElse(25); final int pageStart = (currentPage - 1) * uiSize; final int remaining = Math.max(0, totalCount - pageStart); final int pageCount = Math.min(uiSize, remaining); refreshPageInfo(); return pageCount; } catch (IOException ex) { logger().error(\u0026#34;Error counting\u0026#34;, ex); totalCount = 0; refreshPageInfo(); return 0; } } ); grid.setPageSize(Optional.ofNullable(searchBar.getPageSize()).orElse(25)); grid.setDataProvider(dataProvider); } This makes it clear how closely the grid, SearchBar, and backend interact: the DataProvider calculates the effective query parameters from the current paging information and page size, then builds a UrlMappingListRequest from the SearchBar and passes it directly to the URLShortenerClient. The SearchBar is also used for the counting function, this time without paging parameters, to determine the total number of entries.\nThe advantage of this structure is that the OverviewView itself does not need to know the details of the filter fields. It delegates the entire filter configuration to the SearchBar and focuses solely on controlling the grid and paging. Changes to the filters – such as additional criteria or changed default values – can be made entirely in the SearchBar without having to adjust the DataProvider or the View.\nOverall, the interaction among SearchBar, Grid, and the backend shows a cleanly orchestrated data-flow model: Users change a filter, the SearchBar generates a unique search query, the DataProvider requests the appropriate data via the URLShortenerClient, and the Grid presents the results. This consistent, clearly structured process makes the entire interface much more stable, understandable and responsive.\nCheers Sven\n","date":"21 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-extracting-components-part-1/","section":"Posts","summary":"Today marks a crucial step in the evolution of the URL shortener’s user interface. After the focus in the past few days was mainly on functional enhancements – from filter and search functions to bulk operations – this day is dedicated to a structural fine-tuning: the refactoring of central UI components. This refactoring not only serves to clean up the code but also creates a clear, modular basis for future extensions as well as a significantly improved developer experience.\n","title":"Advent Calendar 2025 - Extracting Components - Part 1","type":"posts"},{"content":" What has happened so far? # In the first part of this article, the new active/inactive model for shortlinks was introduced and anchored at the architectural level. Based on the technical rationale, it was shown that a pure expiration date is insufficient for modern use cases and that an explicit activity status is required.\nBased on this, the technical foundations were laid: the core domain model was extended with an active flag, the DTOs were adapted accordingly, and the serialisation was designed to ensure backward compatibility. In addition, the administrative REST endpoints have been expanded to enable targeted setting, querying, and filtering of activity status. The redirect behaviour has also been clarified so that deactivated and expired shortlinks can be distinguished by clearly defined HTTP status codes.\nThe structural basis of the active/inactive model is thus fully established. In the second part, the focus is on practical use: how the Java client maps these new capabilities, how users can conveniently control activity status via the Vaadin interface, and how this results in consistent, efficient workflows in everyday use.\nThe source code for this project‘s status can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-10\nJava Client Enhancements # After the server API had been extended to include functions for activating and deactivating shortlinks, the Java client also had to be adapted accordingly. After all, it is the primary interface for many users to work programmatically with the URL shortener – whether in the context of desktop tools, automations, CI/CD pipelines or embedded systems.\nChapter 5 details how new capabilities have been added to the client to support the active/inactive model fully. These include:\nthe targeted switching of the activity status, the initial setting of the activity and expiration date when creating a shortlink, as well as editing existing mappings, including the new activity field. The extensions are based on a consistent design approach: simple parameters, clear method focus, strict validation, and traceable error handling. For users, this creates an API that is not only complete, but also intuitive to use – regardless of whether individual values are changed or complete mappings are rebuilt.\nNew API: toggleActive(shortCode, active) # To allow the user to control the activity status of a shortlink not only via the REST API, but also conveniently via the Java client, a new method has been added to the client API. This method is the functional equivalent of the toggle endpoint on the server side and allows shortlinks to be enabled or disabled directly from applications, scripts, or automations.\nThe new API method toggleActive(shortCode, active) does exactly this. It ensures that all relevant information is transmitted to the server in the correct structure and that the server\u0026rsquo;s response is converted into a suitable representation. With a clear focus on switching activity status, the user eliminates the need to build full update objects or send unnecessary data.\nAnother advantage of this method is its simplicity: the user only needs to specify the shortcode of the link to be changed as well as the desired new status. The client\u0026rsquo;s internal logic takes care of everything else – from creating the appropriate request payload to interpreting the server response. This makes it particularly intuitive to use and reduces potential sources of error.\nIn the next step, we will look at the concrete implementation of this method using the source code. The following implementation comes directly from the URLShortenerClient:\npublic boolean toggleActive(String shortCode, boolean active) throws IOException { logger().info(\u0026#34;Toggle Active shortCode=\u0026#39;{}\u0026#39; active=\u0026#39;{}\u0026#39;\u0026#34;, shortCode, active); if (shortCode == null || shortCode.isBlank()) { throw new IllegalArgumentException(\u0026#34;shortCode must not be null/blank\u0026#34;); } final URI uri = serverBaseAdmin.resolve(PATH_ADMIN_TOGGLE_ACTIVE + \u0026#34;/\u0026#34; + shortCode); final URL url = uri.toURL(); logger().info(\u0026#34;Toggle Active - {}\u0026#34;, url); final HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestMethod(\u0026#34;PUT\u0026#34;); con.setDoOutput(true); con.setRequestProperty(CONTENT_TYPE, JSON_CONTENT_TYPE); con.setRequestProperty(ACCEPT, APPLICATION_JSON); con.setConnectTimeout(CONNECT_TIMEOUT); con.setReadTimeout(READ_TIMEOUT); var req = new ToggleActiveRequest(shortCode, active); final String body = toJson(req); logger().info(\u0026#34;Toggle Active - request body - \u0026#39;{}\u0026#39;\u0026#34;, body); try (OutputStream os = con.getOutputStream()) { os.write(body.getBytes(UTF_8)); } final int code = con.getResponseCode(); logger().info(\u0026#34;Toggle Active - responseCode {}\u0026#34;, code); if (code == 200 || code == 204 || code == 201) { drainQuietly(con.getInputStream()); return true; } if (code == 404) { logger().info(\u0026#34;shortCode not found.. {}\u0026#34;, shortCode); drainQuietly(con.getErrorStream()); return false; } final String err = readAllAsString(con.getErrorStream()); throw new IOException(\u0026#34;Unexpected response: \u0026#34; + code + \u0026#34;, body=\u0026#34; + err); } The method starts with basic validation and logging. The shortCode parameter must be set because it uniquely identifies the shortlink to change. If the value is null or empty, the client throws an IllegalArgumentException even before an HTTP call is made.\nThe next step is to create the destination URL for the API call. The server\u0026rsquo;s administrative base path (serverBaseAdmin) is combined with the toggle endpoint path segment. Thanks to this dynamic composition, the client remains flexible in different deployment environments.\nThe client then opens an HTTP connection and configures it for a PUT request. The method sets the expected header fields, including Content-Type (for JSON) and Accept, to define the expected response type. setDoOutput(true) indicates that a request body is included in the request.\nFor the actual payload, an instance of ToggleActiveRequest is created, which consists of a shortCode and the desired new active state. This structure is serialized using toJson and then written to the output stream of the connection.\nAfter the request is sent, the method reads the HTTP status code via con.getResponseCode(). The implementation distinguishes between three main cases:\nSuccessful state change (200, 204, or 201): The method flushes the InputStream via drainQuietly and returns true. This signals to the user that the shortlink has been updated successfully. Shortlink not found (404): Again, the ErrorStream is emptied. However, the method returns false to make it clear to the user that the shortlink does not exist and therefore cannot be updated. All other error cases : In the event of unexpected or erroneous responses, the ErrorStream is read and packaged together with the HTTP status code in an IOException. This forces the calling code to handle unforeseen errors and prevents such states from being silently ignored. Thus, toggleActive provides a clearly defined, robust API for switching activity status via the Java client. It follows the same design principles as the client\u0026rsquo;s other methods: clear validation, consistent error handling, meaningful logging, and lean, JSON-based communication. The implementation thus integrates seamlessly into the existing client architecture and provides a simple yet effective extension of functionality.\nAdvanced createCustomMapping(...) # In addition to the possibility to activate or deactivate existing shortlinks afterwards, the process for creating new shortlinks has also been expanded. To allow users to determine whether a shortlink is active when making it, the createCustomMapping(...) method is used in the Java client.\nBefore this adjustment, a shortlink could be created using only its fundamental properties – shortcode, original URL, and optional expiration date. The activity status was set implicitly or could only be controlled via later processing steps. With the new extension, the user can determine during creation whether a shortlink is active or inactive.\nThe method follows the same principles as the client\u0026rsquo;s other functions: it focuses on clear, simple, and reliable communication with the server. The user only provides the required input values. At the same time, the client takes care of the entire technical processing of the request – from creating the request object to JSON serialisation and interpreting the server response.\nThis expansion allows new use cases to be realised. For example, a shortlink can already be created, but only activated later – for example, synchronously with a release time, a marketing campaign or automatically in CI/CD pipelines. At the same time, the uniform data structure ensures that the activity status of a shortlink is treated consistently both when creating and editing.\nIn the next step, we examine the concrete implementation of this method using the relevant source code sections and analyse how the extended parameters are integrated into the creation process.\nThe extension appears in the URLShortenerClient in the form of two overloaded methods: a simple variant and an extended version with expiration and activity parameters:\npublic ShortUrlMapping createCustomMapping(String alias, String url) throws IOException { logger().info(\u0026#34;Create custom mapping alias=\u0026#39;{}\u0026#39; url=\u0026#39;{}\u0026#39;\u0026#34;, alias, url); return createCustomMapping(alias, url, null, null); } public ShortUrlMapping createCustomMapping(String alias, String url, Instant expiredAtOrNull, Boolean activeOrNull) throws IOException { logger().info(\u0026#34;Create custom mapping alias=\u0026#39;{}\u0026#39; url=\u0026#39;{}\u0026#39; expiredAt=\u0026#39;{}\u0026#39; active=\u0026#39;{}\u0026#39;\u0026#34;, alias, url, expiredAtOrNull, activeOrNull); var result = UrlValidator.validate(url); if (!result.valid()) { throw new IllegalArgumentException(\u0026#34;Invalid URL: \u0026#34; + result.message()); } if (alias != null \u0026amp;\u0026amp; !alias.isBlank()) { var validate = AliasPolicy.validate(alias); if (!validate.valid()) { var reason = validate.reason(); throw new IllegalArgumentException(reason.defaultMessage); } } var shortenRequest = new ShortenRequest(url, alias, expiredAtOrNull, activeOrNull); String body = toJson(shortenRequest); logger().info(\u0026#34;createCustomMapping - body - \u0026#39;{}\u0026#39;\u0026#34;, body); URL shortenUrl = serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL(); logger().info(\u0026#34;connecting to .. shortenUrl {} (custom)\u0026#34;, shortenUrl); HttpURLConnection connection = (HttpURLConnection) shortenUrl.openConnection(); connection.setRequestMethod(\u0026#34;POST\u0026#34;); connection.setDoOutput(true); connection.setRequestProperty(CONTENT_TYPE, JSON_CONTENT_TYPE); try (OutputStream os = connection.getOutputStream()) { os.write(body.getBytes(UTF_8)); } int status = connection.getResponseCode(); logger().info(\u0026#34;Response Code from Server - {}\u0026#34;, status); if (status == 200 || status == 201) { try (InputStream is = connection.getInputStream()) { String jsonResponse = new String(is.readAllBytes(), UTF_8); logger().info(\u0026#34;createCustomMapping - jsonResponse - {}\u0026#34;, jsonResponse); ShortUrlMapping shortUrlMapping = fromJson(jsonResponse, ShortUrlMapping.class); logger().info(\u0026#34;shortUrlMapping .. {}\u0026#34;, shortUrlMapping); return shortUrlMapping; } } if (status == 409) { final String err = readAllAsString(connection.getErrorStream()); throw new IllegalArgumentException(\u0026#34;Alias already in use: \u0026#39;\u0026#34; + alias + \u0026#34;\u0026#39;. \u0026#34; + err); } if (status == 400) { final String err = readAllAsString(connection.getErrorStream()); throw new IllegalArgumentException(\u0026#34;Bad request: \u0026#34; + err); } throw new IOException(\u0026#34;Server returned status \u0026#34; + status); } The simple variant of createCustomMapping serves as a convenience method: it accepts only an alias and a URL and delegates to the extended version, setting expiration date and activity status to zero. This keeps the API lean for simple use cases, while allowing complete control over the shortlink via the overloaded method.\nThe extended method first assumes responsibility for validating the input data. UrlValidator.validate(url) checks whether the specified destination URL meets the expected criteria. If this is not the case, an IllegalArgumentException is thrown with a comprehensible error message. If an alias is set, the alias policy is then checked. This ensures that custom shortcodes comply with the set rules and, for example, do not contain unwanted special characters or prohibited patterns.\nIf the URL and alias are valid, a ShortenRequest is created that includes expiredAtOrNullandactiveOrNull, in addition to URL and alias. In this way, the user can control whether the shortlink has an expiration date and whether it should be initially active or inactive when creating it. The request is then serialised in JSON format and sent to the PATH_ADMIN_SHORTEN endpoint.\nThe HTTP configuration follows the familiar pattern: A POST request is created with a JSON body, the Content-Type is set accordingly, and the body is transmitted via the OutputStream. The server\u0026rsquo;s response is first checked against the HTTP status code. If successful (200 or 201), the client reads the response body, converts it to a ShortUrlMapping object, and returns it to the user.\nTwo special paths are provided for error cases: If the alias reservation fails because the alias is already assigned (409 Conflict), an IllegalArgumentException is thrown with a clear description. General validation errors (400 Bad Request) also throw a meaningful IllegalArgumentException. All other unexpected status codes result in an IOException that includes the exact status code and error message from the server.\nOverall, the advanced createCustomMapping method fits seamlessly into the existing API design. It allows users to create shortlinks in one step with an alias, expiration date, and activity status, combining strict validation with precise, predictable error handling.\nAdjustments to edit(...) – Processing with activity status # In addition to creating new shortlinks, users often need to customise existing listings. This may involve updating the target URL, adjusting an expiration date, or, in the context of the new functionality, activating or deactivating an existing shortlink.\nTo cover these requirements, the existing edit(...) method of the Java client has been extended. It now also supports the optional activity status parameter, so the user no longer needs to change this value via a downstream toggle API; they can control it directly during editing.\nThis approach facilitates workflows in which multiple properties of a shortlink are edited simultaneously. Instead of making various API calls in sequence, the entire change process can be combined into a single edit operation. This means that the API remains efficient, consistent and can be easily integrated into various application scenarios – from manual editing to the UI to automated processes in scripts or backend systems.\nIn the next step, we will discuss the implementation of the extended edit(...) method.\nThe following implementation comes from the URLShortenerClient and shows how the activity state was incorporated into the editing process:\npublic boolean edit(String shortCode, String newUrl, Instant expiresAtOrNull, Boolean activeOrNull) throws IOException { logger().info(\u0026#34;Edit mapping alias=\u0026#39;{}\u0026#39; url=\u0026#39;{}\u0026#39; expiredAt=\u0026#39;{}\u0026#39; active=\u0026#39;{}\u0026#39;\u0026#34;, shortCode, newUrl, expiresAtOrNull, activeOrNull); if (shortCode == null || shortCode.isBlank()) { throw new IllegalArgumentException(\u0026#34;shortCode must not be null/blank\u0026#34;); } if (newUrl == null || newUrl.isBlank()) { throw new IllegalArgumentException(\u0026#34;newUrl must not be null/blank\u0026#34;); } final URI uri = serverBaseAdmin.resolve(PATH_ADMIN_EDIT + \u0026#34;/\u0026#34; + shortCode); final URL url = uri.toURL(); logger().info(\u0026#34;edit - {}\u0026#34;, url); final HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestMethod(\u0026#34;PUT\u0026#34;); con.setDoOutput(true); con.setRequestProperty(CONTENT_TYPE, JSON_CONTENT_TYPE); con.setRequestProperty(ACCEPT, APPLICATION_JSON); con.setConnectTimeout(CONNECT_TIMEOUT); con.setReadTimeout(READ_TIMEOUT); final ShortenRequest req = new ShortenRequest(newUrl, shortCode, expiresAtOrNull, activeOrNull); final String body = toJson(req); logger().info(\u0026#34;edit - request body - \u0026#39;{}\u0026#39;\u0026#34;, body); try (OutputStream os = con.getOutputStream()) { os.write(body.getBytes(UTF_8)); } final int code = con.getResponseCode(); logger().info(\u0026#34;edit - responseCode {}\u0026#34;, code); if (code == 200 || code == 204 || code == 201) { drainQuietly(con.getInputStream()); return true; } if (code == 404) { logger().info(\u0026#34;shortCode not found.. {}\u0026#34;, shortCode); drainQuietly(con.getErrorStream()); return false; } final String err = readAllAsString(con.getErrorStream()); throw new IOException(\u0026#34;Unexpected response: \u0026#34; + code + \u0026#34;, body=\u0026#34; + err); } The method accepts four parameters: the shortcode to change, the new destination URL, an optional expiration date, and an optional activity status. Right from the start, we check whether the necessary fields – especially shortCode and newUrl – are valid. An IllegalArgumentException immediately catches invalid inputs.\nThe request is then sent as a PUT to the corresponding edit endpoint. The payload consists of a ShortenRequest that contains all editable attributes of a shortlink – including the activity specification activeOrNull. This allows the user to update the URL, expiration date, and activity status in a single process.\nThe client then processes the HTTP response. Success cases (200, 201, 204) return true, while a 404 clearly indicates that the shortlink does not exist. Unexpected status codes cause an IOException, which allows the user to provide precise fault diagnoses.\nThis extension makes the edit(...) method fully capable of updating all relevant properties of a shortlink in a single step, including the activity state, which was previously only controllable via a separate toggle endpoint.\nUser interactions and UI logic in the Vaadin interface # After the previous chapters covered server-side REST handlers, client APIs, and persistence, this chapter focuses on the application\u0026rsquo;s UI layer. The OverviewView is the central administration tool for users to search, filter, edit, and manage shortlinks in bulk.\nChapter 6, therefore, sheds light on the most critical user interactions in the frontend, in particular:\nThe interaction of Vaadin-Grid , CallbackDataProvider and dynamic filter parameters the integration of single actions (e.g. activating/deactivating a link by clicking on an icon) Implement more complex bulk operations , including dialogues, error handling, and visual feedback The connection between UI inputs and the REST endpoints via the URLShortenerClient Switching the active state directly in the grid # To allow users to change the active or inactive status of a shortlink not only via the API, but also conveniently in the user interface, the grid of the administration view has been extended. The goal is to create the most direct, intuitive, and low-risk way possible to switch the status of a shortlink without opening separate dialogues or editing masks.\nAt the centre of each table row is a clearly recognisable visual button. With a single click, the user can toggle a shortlink\u0026rsquo;s status between active and inactive. The interface immediately signals the current state of the shortlink and the action that will be taken when clicked.\nThis enlargement has three main objectives:\nSpeed – frequent switching does not require navigation into additional masks. Transparency – the user can see at all times which shortlinks are currently active. Robustness – possible error situations are clearly communicated and do not affect the rest of the application. The core of the implementation lies in configuring the grid and the new \u0026ldquo;Active\u0026rdquo; column. The relevant snippet from the OverviewView looks like this:\ngrid.addComponentColumn(m -\u0026gt; { Icon icon = m.active() ? VaadinIcon.CHECK_CIRCLE.create() : VaadinIcon.CLOSE_CIRCLE.create(); icon.setColor(m.active() ? \u0026#34;var(--lumo-success-color)\u0026#34; : \u0026#34;var(--lumo-error-color)\u0026#34;); icon.getStyle().set(\u0026#34;cursor\u0026#34;, \u0026#34;pointer\u0026#34;); icon.getElement().setProperty(\u0026#34;title\u0026#34;, m.active() ? \u0026#34;Deactivate\u0026#34;: \u0026#34;Activate\u0026#34;); icon.addClickListener(_ -\u0026gt; { boolean newValue = !m.active(); try { urlShortenerClient.toggleActive(m.shortCode(), newValue); Notification.show(\u0026#34;Status updated\u0026#34;, 2000, Notification.Position.TOP_CENTER); safeRefresh(); } catch (Exception ex) { Notification.show(\u0026#34;Error updating active status: \u0026#34; + ex.getMessage(), 3000, Notification.Position.TOP_CENTER); } }); return icon; }) .setHeader(\u0026#34;Active\u0026#34;) .setKey(\u0026#34;active\u0026#34;) .setAutoWidth(true) .setResizable(true) .setSortable(true) .setFlexGrow(0); Instead of a plain text field, a component column is used here, displaying a separate icon for each row. The choice of icon depends directly on the current activity status of the respective shortlink:\nIf m.active() is true , a CHECK_CIRCLE icon is displayed. If m.active()is false , a CLOSE_CIRCLE icon is used. The colour scheme (success for active, error for inactive) allows the user to recognise the state at a glance. In addition, the title attribute on the icon tells you which action is triggered by a click (\u0026ldquo;Activate\u0026rdquo; or \u0026ldquo;Deactivate\u0026rdquo;).\nThe toggle mechanism is implemented in the icon’s click listener. When clicked, the desired new status is first calculated:\nboolean newValue = !m.active(); The Java client is then called, which delegates the state change to the server via the REST API:\nurlShortenerClient.toggleActive(m.shortCode(), newValue); If the call succeeds, the user receives a short, unobtrusive confirmation notification, and the grid is reloaded via safeRefresh(). This means that subsequent changes (e.g. filtering by active status) are also displayed correctly immediately.\nError handling follows the same pattern as in the client API: If an exception occurs – whether due to network problems, unexpected HTTP status codes or server errors – it is communicated in the UI via a clearly visible notification:\n} catch (Exception ex) { Notification.show(\u0026#34;Error updating active status: \u0026#34; + ex.getMessage(), 3000, Notification.Position.TOP_CENTER); } For the user, this means that the active status of a shortlink can be changed directly in the overview with a click. The UI combines clear visual cues (icon, colour, tooltip) with immediate feedback and robust error handling. This reduces the need for separate processing dialogues and makes typical administrative tasks around active/inactive much more efficient.\nFilter by active and inactive status # In addition to the ability to switch a shortlink\u0026rsquo;s active status directly in the grid, the user interface has been enhanced with a precise filter system. This allows the user to search specifically for active, inactive, or all shortlinks – without manual searches or complex queries.\nThe goal is to provide the user with a tool that allows them to control the visibility of entries flexibly. This improves both the clarity of large datasets and the efficiency of routine administrative tasks, such as detecting expired or disabled shortlinks.\nA unified selection element controls the new filter and affects both data retrieval and paging. If the user sets the filter to Active, Inactive, or Not set, the query parameters are updated to load and display only relevant records in the grid.\nCentral to this is the Select field for the active status, which is defined in the OverviewView:\nprivate final Select\u0026lt;ActiveState\u0026gt; activeState = new Select\u0026lt;\u0026gt;(); This UI element is configured in the search bar and displayed alongside other search and paging elements. Initialisation is done in the buildSearchBar() block:\nactiveState.setLabel(\u0026#34;Active state\u0026#34;); activeState.setItems(ActiveState.values()); activeState.setItemLabelGenerator(state -\u0026gt; switch (state) { case ACTIVE -\u0026gt; \u0026#34;Active\u0026#34;; case INACTIVE -\u0026gt; \u0026#34;Inactive\u0026#34;; case NOT_SET -\u0026gt; \u0026#34;Not set\u0026#34;; }); activeState.setEmptySelectionAllowed(false); activeState.setValue(ActiveState.NOT_SET); HorizontalLayout topBar = new HorizontalLayout(globalSearch, searchScope, pageSize, activeState, resetBtn); This gives the user a clearly labeled drop-down selection with three states:\nActive – only active shortlinks Inactive – only inactive shortlinks Not set – no filtering by active status The filter is set to NOT_SET by default , so all shortlinks are displayed first. The ItemLabelGenerator function determines the label displayed in the drop-down for each enum value.\nFor the filter to be functionally effective, the view reacts to changes in the selection field. A corresponding listener is registered in the addListeners() block:\nactiveState.addValueChangeListener(_ -\u0026gt; { currentPage = 1; safeRefresh(); }); As soon as the user changes the active state, the current page is reset to 1 and the data provider is reloaded. This makes the filter changes directly visible and prevents the user from being on an invalid page (for example, if there are fewer entries due to filtering).\nThe actual connection to the REST API is created in the buildFilter(...)method block that creates a UrlMappingListRequest from the UI :\nprivate UrlMappingListRequest buildFilter(Integer page, Integer size) { UrlMappingListRequest.Builder b = UrlMappingListRequest.builder(); ActiveState activeStateValue = activeState.getValue(); logger().info(\u0026#34;buildFilter - activeState == {}\u0026#34;, activeStateValue); if (activeStateValue != null \u0026amp;\u0026amp; activeStateValue.isSet()) { b.active(activeStateValue.toBoolean()); } // ... Other filters (codePart, urlPart, time periods, sorting, paging) if (page != null \u0026amp;\u0026amp; size != null) { b.page(page).size(size); } var filter = b.build(); logger().info(\u0026#34;buildFilter - {}\u0026#34;, filter); return filter; } Here, the enum value of the activeState select is read and, if it is considered \u0026ldquo;set,\u0026rdquo; converted to a Boolean (true for active, false for inactive). The request builder takes over this value and later transmits it to the server by the client (URLShortenerClient) as a query parameter. If the state is NOT_SET, no active value is set, so there is no active-status restriction on the server side.\nTogether, these building blocks form a consistent filter concept:\nThe Select\u0026lt;ActiveState\u0026gt; provides a clear, three-level selection. The ValueChangeListener ensures that filter changes take effect immediately. buildFilter(...) translates the UI selection into a typed request to the backend API. For users, this creates a seamless interaction: Switching from \u0026ldquo;Active\u0026rdquo; to \u0026ldquo;Inactive\u0026rdquo; immediately leads to only the corresponding shortlinks appearing in the grid – without this logic having to be duplicated in the frontend or filtered on the client side.\nMass operations based on active status. # In addition to single editing, the interface offers convenient bulk operations that let users enable or disable multiple shortlinks at once. This is especially helpful in scenarios where large amounts of data need to be managed, such as temporarily shutting down campaign links or reactivating an entire group of previously disabled shortlinks.\nIn this extension, bulk handling has been designed to integrate seamlessly with the existing grid and the integrated selection mechanisms. The user can select as many rows as they want in the grid and then trigger appropriate actions via clearly recognisable buttons in the bulk bar at the bottom of the screen.\nThe process always follows the same pattern:\nThe user marks the desired short links. A confirmation dialogue ensures that the mass change is deliberately triggered. The action is performed for each selected entry, regardless of potential errors. The result is displayed to the user in a compact success/failure overview. This approach enables robust yet user-friendly bulk editing. The interface remains responsive, and error handling ensures that a failure on a single shortlink does not affect the entire operation.\nThe following snippet is taken directly from the OverviewView and shows the central method that enables or disables a set of shortlinks in one pass:\nprivate void bulkSetActive(Set\u0026lt;ShortUrlMapping\u0026gt; selected, boolean activate) { int success = 0; int failed = 0; for (var m : selected) { try { var ok = urlShortenerClient.toggleActive(m.shortCode(), activate); if (ok) { success++; } else { failed++; } } catch (IOException ex) { logger().error(\u0026#34;Toggle active state failed for {}\u0026#34;, m.shortCode(), ex); failed++; } } grid.deselectAll(); safeRefresh(); var actionLabel = activate ? \u0026#34;Activate\u0026#34; : \u0026#34;Deactivate\u0026#34;; Notification.show( actionLabel + \u0026#34; – Success: \u0026#34; + success + \u0026#34; • Failed: \u0026#34; + failed ); } This method is central to mass activation. It receives the shortlinks currently selected in the grid, as well as the target status (activate = true or false). For each entry, the corresponding request is sent to the server API via the Java client. The approach is deliberately designed to be fault-tolerant: a failure of one entry does not affect the rest of the processing.\nThe UI is not updated until the loop is complete. This keeps the operation clear and efficient, even with many shortlinks. The result is displayed to the user in a compact summary.\nThe user triggers this operation via a confirmation dialogue. This is implemented in confirmBulkSetActiveSelected(boolean activate ):\nprivate void confirmBulkSetActiveSelected(boolean activate) { var selected = grid.getSelectedItems(); if (selected.isEmpty()) { Notification.show(\u0026#34;No entries selected\u0026#34;); return; } var verb = activate ? \u0026#34;activate\u0026#34; : \u0026#34;deactivate\u0026#34;; var verbCap = activate ? \u0026#34;Activate\u0026#34; : \u0026#34;Deactivate\u0026#34;; Dialog dialog = new Dialog(); dialog.setHeaderTitle(verbCap + \u0026#34; all \u0026#34; + selected.size() + \u0026#34; short links?\u0026#34;); dialog.add(new Text( \u0026#34;This will \u0026#34; + verb + \u0026#34; all selected short links. \u0026#34; + \u0026#34;They will be \u0026#34; + (activate ? \u0026#34; active\u0026#34; : \u0026#34;inactive\u0026#34;) + \u0026#34; afterwards.\u0026#34; )); Button cancel = new Button(\u0026#34;Cancel\u0026#34;, _ -\u0026gt; dialog.close()); Button confirm = new Button(verbCap + \u0026#34; All\u0026#34;, _ -\u0026gt; { dialog.close(); bulkSetActive(Set.copyOf(selected), activate); }); confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY); dialog.getFooter().add(new HorizontalLayout(cancel, confirm)); dialog.open(); } This dialogue ensures that mass changes are not accidentally triggered. This is especially relevant for deactivating actions, as a deactivated shortlink no longer redirects.\nThe dialogue is activated via the two buttons in the bulk bar:\nbulkActivateBtn.addClickListener(_ -\u0026gt; confirmBulkActivateSelected()); bulkDiActivateBtn.addClickListener(_ -\u0026gt; confirmBulkDeactivateSelected()); The associated wrapper methods are:\nprivate void confirmBulkActivateSelected() { confirmBulkSetActiveSelected(true); } private void confirmBulkDeactivateSelected() { confirmBulkSetActiveSelected(false); } This means that mass editing is fully integrated into the user interface: selection, confirmation, execution and result feedback follow a clear, consistent process model. For users, an intuitive workflow makes administrative tasks much easier.\nRedirect behaviour for end users. # With the introduction of the new active/inactive mechanism and improved handling of expiration times (expiresAt), the URL shortener\u0026rsquo;s behaviour when calling a shortlink changes fundamentally. While users previously received only a redirect or a generic error, the system now provides more granular HTTP status codes that enable precise conclusions about a link\u0026rsquo;s status.\nExpired Shortlinks (expiresAt) → 410 Gone # If a shortlink has an expiration date and has passed, the user must not be redirected to the original URL when accessed. Instead, the system must clearly signal that this shortlink exists, but is no longer valid. This is precisely what the HTTP status code 410 Gone is used for.\nThe status code 410 indicates \u0026ldquo;permanently removed\u0026rdquo; and makes it unmistakably clear that the shortlink was previously active but is now deliberately unavailable. Unlike 404 Not Found, which means that a resource does not exist, 410 conveys a clear semantic meaning: this shortlink has expired and will never be valid again.\nFor the user, this means the call results in clear error behaviour that remains comprehensible. For developers, on the other hand, this mechanism creates transparency, as they can clearly distinguish between three situations:\nA shortlink does not exist → 404 Not Found A shortlink exists, but is disabled → 404 Not Found A shortlink exists and has expired → 410 Gone The consistent use of the 410 status code also enables better monitoring and cleaner automation processes. Systems such as API gateways, SEO tools, crawlers, or CI/CD pipelines can capture the exact behaviour and thus more precisely explain why a redirect does not occur.\nDisabled (active = false) → 404 Not Found # Another central component of the new active/inactive model is how deactivated shortlinks are handled. While expired links are signalled with the HTTP status code 410 Gone , the system deliberately uses a different status code for disabled links: 404 Not Found.\nThis difference is not accidental, but follows clear safety and operational considerations. A deactivated shortlink should behave as if it no longer exists for the end user, although it remains fully present, visible, and manageable internally.\nThis creates a behaviour ideal for maintenance phases, temporary blockades, campaign stops, or safety-related shutdowns. Developers and administrators retain complete control over the dataset without inadvertently revealing internal state information.\nCheers Sven\n","date":"20 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-de-activate-mappings-part-2/","section":"Posts","summary":"What has happened so far? # In the first part of this article, the new active/inactive model for shortlinks was introduced and anchored at the architectural level. Based on the technical rationale, it was shown that a pure expiration date is insufficient for modern use cases and that an explicit activity status is required.\n","title":"Advent Calendar 2025 - De-/Activate Mappings - Part 2","type":"posts"},{"content":" Why an active/inactive model for shortlinks? # For many users – especially those who work in the field of software development – shortlinks are much more than simple URL shorteners. They act as flexible routing mechanisms for campaigns, feature controls, test scenarios, and internal tools. The requirements for transparency, controllability and clean lifecycle management are correspondingly high.\nThe URL shortener receives an active/inactive model that directly supports these claims. A user can now specify, for each shortlink, whether it should be active, temporarily deactivated, or no longer allowed to reach users due to a planned expiration. At the same time, a clear distinction is made between a disabled and an expired shortlink – both in the UI and in the HTTP behaviour.\nFor the user, this creates a noticeable gain in control. An entire campaign can be taken offline with a single click, without losing data or having to edit multiple entries manually. The REST API or the Java client can be used to automatically control activity status externally, enabling integrations with existing development and deployment processes.\nThis introduction outlines the motivation for the new model. The following chapters detail how the UI, API, data model, and redirect logic have been extended, and how users benefit from the new active/inactive mechanism.\nThe source code for this project‘s status can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-10\nArchitecture overview of changes # The new active/inactive model does not work in isolation within a single module; it spans multiple layers of the application. This creates consistent behaviour for the user – regardless of whether a shortlink is edited via the UI, automatically managed via the API or accessed directly in the browser. To understand this interaction, it is worth examining the key areas where adjustments have been made.\nThe basis is the extended data model, which now includes an additional attribute to store the activity status. This feature serves as the common denominator for all shortlink-related operations. As a result, both the REST endpoints and the internal Java client have been extended to reliably transmit activity status when creating, modifying, or querying a shortlink.\nThese structural changes are also reflected in the user interface. The activity status can now be set directly when creating a shortlink, in the overview table, or in the detail dialogue. Actions such as activating or deactivating multiple entries simultaneously use the exact mechanisms as the API, creating a consistent application experience.\nLast but not least, the redirect behaviour has also been adjusted. While expired shortlinks are clearly indicated, deactivated shortlinks are not. In this way, the user can not only better understand why a shortlink is unreachable, but also receive more precise feedback from both client and monitoring perspectives.\nThis architectural overview shows that the active/inactive model is not a one-off feature, but a consistent design principle across all layers of the system. In the following chapters, the respective areas are considered in detail and technically explained.\nThe Advanced Data Model # Before the active/inactive model becomes visible in the user interface, REST API, or redirect behaviour, the basic structure must be defined in the data model. Chapter 3 focuses on this basis and shows how a shortlink\u0026rsquo;s activity status is anchored in the core of the system.\nThe data model is the central place where all relevant information converges into a shortlink. Every user action – be it creating, editing, filtering or forwarding a shortlink – sooner or later falls back on this structure. Therefore, it is crucial to introduce an activity status that is consistently and reliably available.\nThis chapter explains how the new active attribute is mapped to domain objects, how it is transported in DTOs, and the serialisation and backwards-compatibility considerations involved. This clean anchoring in the data model makes the active/inactive model a stable part of the entire application, on which all further technical enhancements are based.\nNew active flag in the core domain model. # The introduction of the active/inactive model begins at the core domain model level. This defines how a shortlink stores its state and what information is transmitted within the application. The new active attribute lets users check at any time whether a shortlink is currently in use or has been deliberately deactivated.\nThe central element of this change is the extension of the ShortUrlMapping class. It outlines the essential characteristics of a short link, including the alias, the original destination URL, and the expiration dates. The new attribute enables the unique determination of the activity status within this structure.\nIn the original implementation, the relevant snippet looks like this:\nprivate boolean active; public boolean active() { return active; } public ShortUrlMapping withActive(boolean active) { return new ShortUrlMapping( this.shortCode, this.originalUrl, this.createdAt, this.expiresAt, active ); } This enhancement ensures that activity status is reliably available both in memory and in all downstream layers of the system. The model remains unchanged. The status is not changed in an existing object; instead, it is created as a new instance using the withActive method. This prevents unintended side effects and supports consistent, deterministic data behaviour.\nThis clear separation between active and inactive states provides a stable foundation for UI, server, and redirect logic. In the following subchapters, we will discuss how this information is mapped to the DTOs and the consequences for serialisation and transport behaviour.\nImpact on DTOs (ShortenRequest, ShortUrlMapping) # The active/inactive model not only affects the domain core but must also be transported cleanly across the application boundary. Data Transfer Objects (DTOs) are crucial for this. They serve as the interface between the user interface, REST API, and Java client, ensuring that the activity status of a shortlink is transmitted correctly.\nWith the expansion of the data model, the DTO ShortUrlMapping also receives an additional attribute that reflects the status of a shortlink. This value is used both in communication from the server to the UI and between the server and the client. A relevant excerpt from the original implementation looks like this:\npublic record ShortUrlMapping( String shortCode, String originalUrl, Instant createdAt, Instant expiresAt, boolean active ) {} This extended DTO enables the user to view activity status in the UI and allows external systems to read the status and respond accordingly. By using a record, structure and serialisation remain lean and clearly defined.\nThe request to create or change a shortlink has also been adjusted. The ShortenRequest now also includes information on whether a shortlink should be made directly active or initially deactivated. This allows the user to control how the shortlink behaves during creation.\nAn excerpt from the corresponding record:\npublic record ShortenRequest( String shortCode, String originalUrl, Instant expiresAt, Boolean active ) {} At this point, it is evident that the request uses a Boolean, whereas the mapping itself uses a primitive Boolean. This difference is deliberate: the value in the request may also be null if the user does not want to set an explicit state. In this case, the server logic decides on a default value. Mapping, on the other hand, always has a defined state, eliminating ambiguity.\nExtending DTOs ensures the active/inactive model is uniformly available across all system boundaries. It forms the basis for the UI, API and client to be informed identically about the status of a shortlink, thus creating a consistent user experience.\nSerialisation and backward compatibility # For the active/inactive model to function reliably, the new active value must not only be present in the internal data model but also correctly serialised and transported across different components. The extension of serialisation affects two areas in particular: JSON communication via REST and data exchange between the various application modules.\nBy using Java records in the DTOs, serialisation is primarily handled by the chosen JSON library. Once the active attribute is defined in the record constructors, it is automatically included in the JSON representation. This makes integration straightforward and minimises the need for additional configuration.\nAn example of the resulting JSON structure of a shortlink as served via the REST API:\n{ \u0026#34;shortCode\u0026#34;: \u0026#34;abc123\u0026#34;, \u0026#34;originalUrl\u0026#34;: \u0026#34;https://example.com\u0026#34;, \u0026#34;createdAt\u0026#34;: \u0026#34;2025-05-20T10:15:30Z\u0026#34;, \u0026#34;expiresAt\u0026#34;: \u0026#34;2025-06-01T00:00:00Z\u0026#34;, \u0026#34;active\u0026#34;: true } The JSON structure makes it clear that the activity state is immediately visible to the user and can be retrieved without any additional logic. This increase in transparency is central to the new model.\nAn essential aspect of this change is backward compatibility. Systems or components that use older versions of the API and are not aware of the active field can continue to interact without issues. JSON consumers typically ignore unknown fields to avoid affecting older clients. At the same time, the server can set a sensible default value for requests that do not specify an active value – typically true.\nThis combination of clear serialisation and high backward compatibility ensures that the active/inactive model can be integrated into existing systems without disruption. At the same time, further development remains open to future extensions, such as more differentiated activity states or audit information.\nEnhancements to the Admin REST API # The active/inactive model only unfolds its full effect through the extensions of the administrative REST API. This establishes the connection between the data model, the internal business logic, and the various clients that create, edit, or automatically manage shortlinks. To enable users to effectively use the new activity state, several API endpoints had to be adapted or supplemented.\nChapter 4 examines these extensions in detail. It shows how the new toggle endpoint enables targeted switching of activity status, how inactive entries can be queried via a dedicated list endpoint, and how the ActiveState filter extends existing query operations. It also explains how differentiated HTTP status codes clearly convey the semantic meaning of a deactivated or expired shortlink.\nThese API customisations create a coherent, well-integrated interface that can map all aspects of the active/inactive model, whether the requests come from a user interface, a Java client, or an automated system.\nNew Toggle Endpoint # To make the active/inactive shortlink model usable across the application, the server component needs a well-defined mechanism to change the activity status of an existing shortlink. For this purpose, a new toggle endpoint was introduced and implemented on the server side by ToggleActiveHandler. It forms the basis for allowing users to adjust the activity state of a shortlink directly from the REST API – whether from the user interface, automations, or external systems.\nThe new endpoint has a clear purpose: it enables the targeted switching of the activity status of a specific shortlink. The user can specify, via a simple request, whether a shortlink should be activated or deactivated without changing any other mapping properties. This not only makes the operation more efficient, but also less error-prone, as there is no need for full update payloads.\nThis endpoint integrates seamlessly with the existing API structure and uses the same transport models and validation mechanisms as other administrative operations. At the same time, it provides a clear separation between changes of activity status and other editing operations, simplifying both implementation and use.\nThe core component of this endpoint is the ToggleActiveHandler. It encapsulates the HTTP-specific processing and delegates the actual state change to the UrlMappingStore behind it:\npublic class ToggleActiveHandler implements HttpHandler, HasLogger { private final UrlMappingStore store; public ToggleActiveHandler(UrlMappingStore store) { this.store = store; } @Override public void handle(HttpExchange ex) throws IOException { logger().info(\u0026#34;handle ... {} \u0026#34;, ex.getRequestMethod()); if (! RequestMethodUtils.requirePut(ex)) return; try { final String body = readBody(ex.getRequestBody()); ToggleActiveRequest req = fromJson(body, ToggleActiveRequest.class); var shortCode = req.shortCode(); if (isNullOrBlank(shortCode)) { writeJson(ex, BAD_REQUEST, \u0026#34;Missing \u0026#39;shortCode\u0026#39;\u0026#34;); return; } boolean newActiveValue = req.active(); logger().info(\u0026#34;Toggling active status for {} to {}\u0026#34;, shortCode, newActiveValue); Result\u0026lt;ToggleActiveResponse\u0026gt; mapping = store.toggleActive(shortCode, newActiveValue); if (mapping.isPresent()) { logger().info(\u0026#34;Toggling active status successfully\u0026#34;); writeJson(ex, OK, toJson(mapping.get())); } else { Mapping. ifFailed(failed -\u0026gt; logger().info(\u0026#34;Toggling active status failed: {}\u0026#34;, failed)); writeJson(ex, BAD_REQUEST, \u0026#34;Toggling active status failed\u0026#34;); } } catch (RuntimeException e) { logger().warn(\u0026#34;catch - {}\u0026#34;, e.toString()); writeJson(ex, INTERNAL_SERVER_ERROR); } finally { logger().info(\u0026#34;ToggleActiveHandler .. finally\u0026#34;); ex.close(); } } } The handler implements the HttpHandler interface and integrates project-specific logging via HasLogger. In the constructor, the dependency is injected into the UrlMappingStore, which is later used to execute the logic that changes the state.\nThe main logic is in the try block. First, the request body is completely read in via readBody(ex.getRequestBody()) and converted to an instance of ToggleActiveRequest using fromJson. This Request object contains the two relevant pieces of information: the shortCode and the new active value. It then checks whether the shortcode is empty or null. In this case, the handler responds immediately with a 400 Bad Request and a clear error message, preventing the store layer from encountering invalid data.\nIf the shortcode is valid, the desired new activity value is extracted and logged. The actual status change occurs via \u0026ldquo;store.toggleActive(shortCode, newActiveValue)\u0026quot;. The result is encapsulated in a Result\u0026lt;ToggleActiveResponse\u0026gt; to ensure both success and failure cases are handled consistently.\nIn case of success (mapping.isPresent()), another success entry is written to the log, and the updated display of the shortlink is returned to the user as JSON with the HTTP status 200 OK. If the store does not return a result, the cause of the error is logged via ifFailed, and the user is sent a response with a 400 Bad Request status code and a generic error message. This clearly defines the error behaviour without revealing too many details internally.\nIf unexpected runtime errors occur in the handler itself, the catch block catches the exception, logs a warning, and returns a 500 Internal Server Error. The finally block ensures that the connection is closed cleanly via ex.close() in all cases and that a corresponding log entry is created. Overall, this creates a robust, clearly structured endpoint that reliably and comprehensibly updates a shortlink‘s activity status.\nQuery inactive links # In addition to targeting the activity status of an individual shortlink, the user needs the ability to view collected inactive entries. This provides an overview of shortlinks that have been deliberately or automatically disabled and are currently no longer redirecting.\nFor this purpose, a special API endpoint was introduced that returns only inactive shortlinks. This endpoint complements the general list endpoint and provides a clear, filtered view of all shortlinks with the \u0026ldquo;inactive\u0026rdquo; status.\nThe focus of this endpoint is on visibility and control. Users can quickly see which shortlinks are currently disabled without having to implement their own filtering logic or evaluate the complete list of shortlinks. This is a notable advantage, especially in larger installations or automated workflows, as inactive entries often require separate management processes.\nThe endpoint integrates seamlessly with the existing administrative API structure and uses the same data models and return formats as other query operations. It ensures that information about inactive shortlinks is consistently retrievable across all UIs and clients.\nThe technical implementation of querying inactive shortlinks is integrated into the existing list handler. The following excerpt shows the ListHandler responsible for this, which provides different list variants, including the inactive entries, via a common endpoint:\npublic class ListHandler implements HttpHandler, HasLogger { private final UrlMappingLookup store; public ListHandler(UrlMappingLookup store) { this.store = store; } SNIPP @Override public void handle(HttpExchange ex) throws IOException { if (! RequestMethodUtils.requireGet(ex)) return; final String path = ex.getRequestURI().getPath(); logger().info(\u0026#34;List request: {}\u0026#34;, path); String responseJson; if (path.endsWith(PATH_ADMIN_LIST_ALL)) { responseJson = listAll(); } else if (path.endsWith(PATH_ADMIN_LIST_EXPIRED)) { responseJson = listExpired(); } else if (path.endsWith(PATH_ADMIN_LIST_ACTIVE)) { responseJson = listActive(); } else if (path.endsWith(PATH_ADMIN_LIST_INACTIVE)) { responseJson = listInActive(); } else if (path.endsWith(PATH_ADMIN_LIST)) { responseJson = listFiltered(ex); } else { logger().info(\u0026#34;undefined path {}\u0026#34;, path); ex.sendResponseHeaders(404, -1); return; } writeJson(ex, OK, responseJson); } private String listInActive() { final Instant now = Instant.now(); return filterAndBuild(\u0026#34;inactive\u0026#34;, m -\u0026gt; { if (!m.active()) return true; return m.expiresAt() .map(exp -\u0026gt; exp.isBefore(now)) .orElse(false); }); } private String filterAndBuild(String mode, Predicate\u0026lt;ShortUrlMapping\u0026gt; predicate) { final Instant now = Instant.now(); final var data = store .findAll() .stream() .filter(predicate) .map(m -\u0026gt; toDto(m, now)) .toList(); return JsonUtils.toJsonListing(mode, data.size(), data); } private Map\u0026lt;String, String\u0026gt; toDto(ShortUrlMapping m, Instant now) { boolean expired = m. expiresAt() .map(t -\u0026gt; t.isBefore(now)) .orElse(false); return Map.of( \u0026#34;shortCode\u0026#34;, m.shortCode(), \u0026#34;originalUrl\u0026#34;, m.originalUrl(), \u0026#34;createdAt\u0026#34;, m.createdAt().toString(), \u0026#34;expiresAt\u0026#34;, m.expiresAt().map(Instant::toString).orElse(\u0026#34;\u0026#34;), \u0026#34;active\u0026#34;, m.active() + \u0026#34;\u0026#34;, \u0026#34;status\u0026#34;, expired ? \u0026#34;expired\u0026#34; : \u0026#34;active\u0026#34; ); } } The ListHandler, like the toggle handler, implements the HttpHandler interface and uses HasLogger for consistent logging. A UrlMappingLookup is injected into the constructor and used for the data queries. The handle method handles routing based on the request path: Depending on whether the request ends in PATH_ADMIN_LIST_ALL, PATH_ADMIN_LIST_ACTIVE, or PATH_ADMIN_LIST_INACTIVE, a specialised list method is called.\nThe listInActive() method is responsible for querying inactive shortlinks. It determines the current time and calls filterAndBuild(\u0026quot;inactive\u0026quot;, ...) with a predicate that selects the desired entries. A shortlink is considered inactive if it is either explicitly marked as inactive (!m.active()) or if it is active but has already expired. The combination of the active flag and the expiration date allows a clean separation between currently usable and no longer usable entries.\nThe filterAndBuild method reads all entries from the UrlMappingLookup, applies the passed predicate, and converts the remaining mappings into a DTO-like structure. For this purpose, toDto is used to create a map of the most essential attributes from each ShortUrlMapping. In addition to shortCode, originalUrl, createdAt, and expiresAt, the active status and a derived field, status, are set here, which distinguishes between expired and active. This additional information makes it easier for the user to read the entry status directly from the response.\nThis structure allows queries for inactive links to fit seamlessly into the existing list concept. Instead of introducing a completely independent endpoint, the existing infrastructure is used and expanded to include a targeted view of inactive entries. The result is a precise, reusable mechanism that always gives users a complete overview of all currently inactive shortlinks.\nAdjustments in the list request (ActiveState filter) # To allow users to search specifically for active, inactive, or all shortlinks, a dedicated activity status filter has been added to the existing list mechanism. This so-called ActiveState filter complements the previous filter criteria, such as text search, sorting, time periods, or pagination, and enables precise control of the search results via the API.\nThis extension provides the user with a flexible yet clearly defined way to set the desired status directly via a query parameter. Instead of retrieving the entire list of shortlinks and then filtering it on the client side, the request can now be restricted directly on the server. This saves resources, reduces unnecessary data transfers, and ensures consistent search results.\nThe ActiveState filter considers three possible states:\nActive – The shortlink is active and, if there is an expiration date, has not yet expired. Inactive - The shortlink has been disabled by the user or has expired. Not set – The user does not provide a status indication, so the filter is not applied, and all relevant entries are considered. This distinction is communicated via the active query parameter, which is passed to the standard list endpoint in a GET request. The server evaluates this parameter and creates a corresponding UrlMappingFilter object based on it, which controls entry selection.\nThe technical implementation of this filter logic is handled in the existing ListHandler, which already handles filtering and sorting shortlinks. The following excerpt shows how to process the active parameter and embed it in the UrlMappingFilter:\nvar filter = UrlMappingFilter.builder() .codePart(first(query, \u0026#34;code\u0026#34;)) .urlPart(first(query, \u0026#34;url\u0026#34;)) .createdFrom(parseInstant(first(query, \u0026#34;from\u0026#34;), true).orElse(null)) .createdTo(parseInstant(first(query, \u0026#34;to\u0026#34;), false).orElse(null)) .active(parseBoolean(first(query, \u0026#34;active\u0026#34;)).orElse(null)) .offset(offset) .limit(size) .sortBy(sortBy.orElse(null)) .direction(dir.orElse(null)) .build(); The central point is the line:\n.active(parseBoolean(first(query, \u0026#34;active\u0026#34;)).orElse(null)) Here, the query parameter active is read, interpreted as an optional boolean. The process in detail:\n**first(query, \u0026quot;active\u0026quot;)** reads the first value of the query parameter active – something like: active=true active=false or the parameter is missing completely. **parseBoolean(...)** converts this value to an Optional\u0026lt;Boolean\u0026gt; . This means that invalid or missing values can also be caught cleanly. **.orElse(null)** ensures that if input is missing or cannot be interpreted, the value null is transferred to the filter. This results in the following meaning:\nactive=true → only active shortlinks active=false → only inactive shortlinks no parameter → no restriction The UrlMappingLookup then processes the resulting UrlMappingFilter:\nint total = store.count(filter); var results = store.find(filter); These methods replace the application of the filter to the saved shortlinks. By passing through the active state, the lookup layer can precisely select the entries corresponding to the desired activity state.\nFinally, the results are transferred to the DTO structure and returned as a paginated JSON response:\nvar items = results.stream().map(m -\u0026gt; toDto(m, now)).toList(); return toJsonListingPaged(\u0026#34;filtered\u0026#34;, items.size(), items, page, size, total, sortBy.orElse(null), dir.orElse(null)); With this extension, the ActiveState filter fits seamlessly into the existing filter system. Users can now target active or inactive shortlinks without additional endpoints or separate calls. This shows how ActiveState is processed and how it affects the final JSON result.\nHTTP status code semantics: 410 vs. 404 # With the introduction of the active/inactive model, the behaviour of the forwarding logic also changes. For the user, the state of a shortlink must be clearly and unambiguously communicated to the outside world – especially if a redirect cannot be done as expected. For this reason, the system uses two distinct HTTP status codes to distinguish expired and disabled shortlinks.\nA shortlink can no longer be reached for two reasons:\nIt has expired. The expiration date has passed, and the shortlink is no longer valid as planned. It has been disabled. The user has manually disabled the shortlink, regardless of the expiration date. Although both result in no redirect, they differ in meaning. Expired shortlinks are intended to signal that their lifespan is ending clearly. Disabled shortlinks, on the other hand, have been deliberately taken out of service and may be in a temporary state.\nTwo HTTP status codes represent this semantic separation:\n410 Gone – for expired short links. This status indicates that the resource is permanently unavailable and will not become available again. 404 Not Found – for disabled shortlinks. Although the shortlink exists, its redirect is intentionally not carried out. The 404 status code indicates a temporary or permanent error. This distinction provides users, API clients, and monitoring systems with more precise feedback on the shortlink\u0026rsquo;s state. Understandably, a shortlink is not accessible due to a natural process or a manual decision.\nCheers Sven\n","date":"19 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-de-activate-mappings-part-1/","section":"Posts","summary":"Why an active/inactive model for shortlinks? # For many users – especially those who work in the field of software development – shortlinks are much more than simple URL shorteners. They act as flexible routing mechanisms for campaigns, feature controls, test scenarios, and internal tools. The requirements for transparency, controllability and clean lifecycle management are correspondingly high.\n","title":"Advent Calendar 2025 - De-/Activate Mappings - Part 1","type":"posts"},{"content":" What has happened so far? # In the first part of \u0026ldquo;Basic Login Solution\u0026rdquo;, the foundation for a deliberately simple yet structurally clean admin login was laid. The starting point was the realisation that even a technically lean URL shortener requires a clear separation between public-facing functions and administrative operations. The goal was not complete user management, but rather a minimal access barrier that integrates seamlessly with the existing Java and Vaadin architectures.\nThe central component of this solution is a lean, file-based configuration in auth.properties. With just two parameters – an activation switch and a password – the login can be fully controlled. The associated LoginConfigInitializer ensures that this configuration is read when the servlet container starts and remains consistently available to the application. This clearly defines whether and how the protection mechanism takes effect even before any UI is rendered.\nBased on this, an independent login view was introduced that does not require MainLayout. It serves as a clear entry point into the administrative area and separates the login context from the rest of the UI, both visually and technically. The implementation focuses on a simplified user experience: a password field, a clear call to action, and clear feedback on success or failure. At the same time, an important architectural principle of today is already evident here: security logic and UI interaction remain neatly separated.\nAfter completing Part 1, a functional login is available, but no full access control is yet in place. Without additional measures, unauthenticated users could continue to access administrative views directly, for example, via deep links or saved URLs. This is precisely where the second part comes in.\nIn Part 2, login is effectively enforced for the first time: via central route protection in MainLayout, consistent session management with SessionAuth, and a clean logout mechanism. This transforms an isolated login screen into a complete, end-to-end authentication flow that reliably protects the application\u0026rsquo;s administrative area.\nThe source code for this article can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-09\nRoute Protection \u0026amp; Access Logic # With the introduction of admin login, it is not enough to provide only a login page. The decisive factor in the protection\u0026rsquo;s effectiveness is that all administrative views are consistently protected and accessible only to authenticated users. This is precisely where Route Protection comes in: in conjunction with LoginConfig and SessionAuth, it ensures the application clearly distinguishes between logged-in and non-logged-in users.\nInstead of securing each view individually, the implementation uses a single central location: the MainLayout. Since all relevant management views use this layout, it is a natural anchor point for bundling the access logic. As soon as a view with this layout is loaded, the MainLayout can check whether login is enabled and whether the current user is already authenticated. Only when these conditions are met is access to the target view granted.\nThere are several benefits to implementing route protection within the layout. On the one hand, the code in individual views remains lean because they do not have to perform security checks themselves. On the other hand, a uniform logic is implemented that applies equally to all protected areas. If the mechanism is later expanded or adapted, a change at a central location can affect behaviour across the entire application.\nFrom a functional perspective, route protection follows a simple process: when a navigation enters a view that uses MainLayout, it checks whether login is globally enabled before rendering. If this is not the case, the application behaves as before and still allows direct access to all pages. If login is enabled, the logic checks whether the current session is marked as authenticated. If not, the user will be automatically redirected to the login page.\nAn exception is the login view itself. It must always be accessible without prior authentication, as it enables access to the protected area. If they were also subject to route protection, an infinite redirect loop would result. The implementation explicitly handles this special case by excluding the login route from the check.\nIn practice, this mechanism results in transparent, predictable behaviour: non-logged-in users who open a deep link directly into the admin UI are automatically redirected to the login page. After a successful login, you will be taken to the desired administration page and can navigate the protected area without further interruptions. At the same time, the option to altogether turn off the login at the configuration level is retained – in this case, the application behaves as if route protection never existed.\nImplementation in MainLayout # The technical implementation of route protection is anchored in the MainLayout. This not only serves as a visual framework for the administration interface but also assumes a central control function for all navigation leading to the protected area via the BeforeEnterObserver interface.\nFirst, the layout is extended so that it can participate in navigation and, at the same time, receive logging functionality:\nextends AppLayout implements BeforeEnterObserver, HasLogger { The implementation of BeforeEnterObserver enables the MainLayout to perform a check before each navigation to a View that uses it. At the same time, HasLogger provides a convenient logging API to make decisions and states in the log traceable.\nThe actual route protection is bundled in thebeforeEnter method:\n@Override public void beforeEnter(BeforeEnterEvent event) { If login is globally turned off, do not protect any routes. if (! LoginConfig.isLoginEnabled()) { return; } logger().info(\u0026#34;beforeEnter target={} authenticated={}\u0026#34;, event.getNavigationTarget().getSimpleName(), SessionAuth.isAuthenticated()); The login view itself must never be protected, otherwise we create a loop if (event.getNavigationTarget().equals(LoginView.class)) { return; } if (! SessionAuth.isAuthenticated()) { logger().info(\u0026#34;beforeEnter.. isAuthenticated()==false - reroute to LoginView\u0026#34;); event.rerouteTo(LoginView.class); } } The logic follows the process described above exactly. First, it is checked whether the login is active per the configuration. If login.enabled is set to false in the auth.properties file, the method returns immediately without any restrictions. In this mode, the application behaves as it did before Route Protection was introduced.\nIf login is enabled, the next step is to check the current navigation destination and verify whether the user is already authenticated—a short log entry records which view to navigate to and the current authentication status. If the target is the LoginView, the check is aborted – the login page always remains accessible, regardless of whether the session is already logged in or not.\nFor all other views, if the session is not marked as logged in by SessionAuth.isAuthenticated(), the user will be transparently redirected to the login page. The original target view is not rendered, so no administrative functions are visible without authentication.\nLogout button in the header # Closely linked to route protection is the ability to clean end an existing session. This functionality is also implemented in the MainLayout and appears to the user as a logout button in the header. The creation of this button is linked to the configuration:\nHorizontalLayout headerRow; if (LoginConfig.isLoginEnabled()) { var logoutButton = new Button(\u0026#34;Logout\u0026#34;, _ -\u0026gt; { SessionAuth.clearAuthentication(); UI.getCurrent().getPage().setLocation(\u0026#34;/\u0026#34; + LoginView.PATH); }); logoutButton.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE); headerRow = new HorizontalLayout(toggle, appTitle, new Span(), storeIndicator, logoutButton); } else { headerRow = new HorizontalLayout(toggle, appTitle, new Span(), storeIndicator); } If the login is deactivated, no logout button is displayed because there is no state to terminate. If the login is active, MainLayout adds a simple inline logout button to the header. Clicking it calls SessionAuth.clearAuthentication(), removes the authentication attribute from the VaadinSession, and closes the session. The browser is then explicitly redirected to the login page via setLocation.\nThe combination of the beforeEnter check and the logout button creates consistent access logic: Users are directed to the login page when they first access the protected area, can move freely within the admin UI after successful login, and can end their session at any time in a visible, controlled manner.\nIn the next chapter, we will focus on the SessionAuth class, which encapsulates the login status in the session and provides a centralised access point.\nLogout function in the header # A login mechanism is only complete if it also knows a clear way back. In practice, this means that anyone who logs in to the admin interface must be able to end their session just as consciously. The introduction of a logout function is therefore not only a technical addition but also an essential signal to users that access to the administrative area is considered a sensitive context.\nIn the URL shortener, this functionality is shown as a slim logout button in the header. It is visible only if login protection is enabled in the configuration and the application is in \u0026ldquo;protected mode\u0026rdquo;; in all cases where \u0026ldquo;login.enabled=false\u0026quot;, the UI does not display this button because there is no session to log off. This deliberately keeps the interface tidy in simple development or demo scenarios.\nFrom the user\u0026rsquo;s perspective, the logout button is inconspicuously integrated into the existing header. It does not appear as a prominent CTA, but it is always available in the admin area. The naming is deliberately kept clear: \u0026ldquo;Logout\u0026rdquo; leaves no room for interpretation and signals that the current authorisation context is ended here. In environments where multiple people work one after the other using the same browser and admin interface, this clarity is essential.\nTechnically, clicking the logout button performs two tasks: first, the authentication status in the current VaadinSession is reset; second, the browser is redirected to the login page. This ensures that the administrative views do not remain open in the browser; subsequent access still follows the login flow. Even an accidentally open tab loses its authorisation as soon as the user logs out.\nThe implementation adheres closely to the rest of the login architecture. The button does not check itself whether a session is authenticated; instead, it delegates this to the small helper class SessionAuth, which is also used elsewhere, such as for route protection. This keeps the logout consistent: the same abstraction that determines whether a session is considered logged in is also responsible for deleting this status.\nFrom a UX perspective, the logout function also helps structure users\u0026rsquo; mental model. As long as they are logged in, they are in an active administration context where changes to shortlinks and configurations are expected. When they log out, they deliberately revert to a neutral role, in which they use the generated URLs at most from the user\u0026rsquo;s perspective. This separation is particularly helpful in heterogeneous teams where not all participants need administrative rights simultaneously.\nImplementation in MainLayout # The logout function is centrally integrated into the MainLayout. The header is built there, and the logout button can be added alongside the app title and the store indicator. Whether this button is visible depends directly on the login configuration:\nHorizontalLayout headerRow; if (LoginConfig.isLoginEnabled()) { var logoutButton = new Button(\u0026#34;Logout\u0026#34;, _ -\u0026gt; { SessionAuth.clearAuthentication(); UI.getCurrent().getPage().setLocation(\u0026#34;/\u0026#34; + LoginView.PATH); }); logoutButton.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE); headerRow = new HorizontalLayout(toggle, appTitle, new Span(), storeIndicator, logoutButton); } else { headerRow = new HorizontalLayout(toggle, appTitle, new Span(), storeIndicator); } In activated mode, a \u0026ldquo;Logout\u0026rdquo; button is created and added to the header\u0026rsquo;s HorizontalLayout. The look is based on the rest of the UI and uses the LUMO_TERTIARY_INLINE variant, so the button appears discreet and blends visually with the header.\nThe actual logoff logic is in the click listener: First, SessionAuth.clearAuthentication() is called to reset the current session\u0026rsquo;s authentication state. The browser is then explicitly redirected to the login view path. This means that the user is not just taken to a \u0026ldquo;blank\u0026rdquo; page after logging out, but is clearly recognisable back to the entry point of the admin area.\nIf the login function is globally deactivated, the button is omitted. The headerRow layout then consists only of toggle, title, spacer and StoreIndicator. This deliberately keeps the UI slim in scenarios without login protection and does not display a non-working logout button.\nSessionAuth – Session state management # The SessionAuth helper class includes all accesses to the authentication state within the VaadinSession. It is used by both the route protection and the logout button and thus represents the central abstraction layer for login decisions:\npackage com.svenruppert.urlshortener.ui.vaadin.security; import com.svenruppert.dependencies.core.logger.HasLogger; import com.vaadin.flow.server.VaadinSession; import static com.svenruppert.dependencies.core.logger.HasLogger.staticLogger; import static java.lang.Boolean.TRUE; public final class SessionAuth implements HasLogger { private static final String ATTR_AUTH = \u0026#34;authenticated\u0026#34;; private SessionAuth() { } public static boolean isAuthenticated() { VaadinSession session = VaadinSession.getCurrent(); if (session == null) return false; var attribute = session.getAttribute(ATTR_AUTH); staticLogger().info(\u0026#34;isAuthenticated.. {}\u0026#34;, attribute); return TRUE.equals(attribute); } public static void markAuthenticated() { staticLogger().info(\u0026#34;markAuthenticated.. \u0026#34;); VaadinSession session = VaadinSession.getCurrent(); if (session != null) { session.setAttribute(ATTR_AUTH, TRUE); } } public static void clearAuthentication() { staticLogger().info(\u0026#34;clearAuthentication.. \u0026#34;); VaadinSession session = VaadinSession.getCurrent(); if (session != null) { session.setAttribute(ATTR_AUTH, null); session.close(); } } } The markAuthenticated() method is called after a successful login and sets the simple attribute authenticatedtoTRUE in the current VaadinSession. With isAuthenticated(), this state can be queried again later – for example, in the route protection of the MainLayout. Both methods use logging to quickly determine when a session was logged in or logged out, in the event of an error, or when analysing user behaviour.\nFor the logout, clearAuthentication() is crucial. It removes the attribute from the session and also closes the VaadinSession. This will also discard other session-related data, and an accidentally opened browser tab will lose its permissions. The next time it is accessed, a new, unauthenticated session is established and guided through the login flow.\nThis tight integration of MainLayout and SessionAuth creates a robust, yet manageable, logout implementation. The header provides a clearly visible exit from the admin context, and the session logic ensures this exit is implemented consistently from a technical standpoint.\nIn the next chapter, we will take a closer look at the SessionAuth class and examine its role in interacting with route protection and the login view within the overall application flow.\nSessionAuth: Session-based authentication in the overall flow # Now that login protection, the login page, and the logout function have been introduced, a central question remains: Where is it actually recorded whether a user is authenticated? The answer lies in the small but crucial helper class SessionAuth, which anchors the authentication state in VaadinSession.\nAt its core, SessionAuth fulfils three tasks. It can first check whether a current session is already registered. Second, after successful login, it can mark the session as authenticated. And thirdly, it can remove the authentication status when logging out and close the session. These three operations form the basis of the login view, route protection in the MainLayout, and the logout button in the header, all of which access a standard truth value rather than maintaining their own state.\nThe chosen approach leverages the fact that Vaadin maintains a separate VaadinSession for each user. This session exists on the server side and is therefore less susceptible to client-side manipulation than, for example, a simple browser flag. By setting a dedicated attribute – such as \u0026ldquo;authenticated\u0026rdquo; – in this session, SessionAuth clearly links the login state to the current browser session. If the same user opens a new browser window, a new session is created; the user is not logged in again.\nIn the interaction between components, this results in a precise flow: if a user reaches the login page and enters a correct password, LoginView calls SessionAuth.markAuthenticated() after a successful check. It then navigates to the administrative overview. As soon as the user opens additional views from there, the MainLayout checks whether the session is still considered logged in, as part of the route protection, using SessionAuth.isAuthenticated(). If this is the case, navigation is allowed. Otherwise, the request will be redirected to the login page.\nFinally, if the user clicks the logout button in the header, SessionAuth.clearAuthentication() removes the authentication attribute from the session and closes VaadinSession. For the server, this session is over. The following request establishes a new session, which is again considered unauthenticated from the application\u0026rsquo;s perspective, so Route Protection redirects the request back to the login view.\nThis session approach fits well with the project\u0026rsquo;s objective: it is deliberately kept simple, requires no additional infrastructure and is easy to understand. At the same time, it meets the requirements of a minimalist admin login, which is less about highly secured, distributed authentication procedures and more about a clear separation between \u0026ldquo;logged in\u0026rdquo; and \u0026ldquo;not logged in\u0026rdquo; within a running browser session.\nIn the rest of this chapter, we will go through the implementation of the three methods isAuthenticated, markAuthenticated and clearAuthentication and look at how they interact at the different points in the code – Login-View, MainLayout and Logout-Button.\nThe implementation of SessionAuth in detail # The SessionAuth class is deliberately kept compact. It encapsulates access to the VaadinSession and provides three static methods, each of which fulfils a clearly defined task:\npackage com.svenruppert.urlshortener.ui.vaadin.security; import com.svenruppert.dependencies.core.logger.HasLogger; import com.vaadin.flow.server.VaadinSession; import static com.svenruppert.dependencies.core.logger.HasLogger.staticLogger; import static java.lang.Boolean.TRUE; public final class SessionAuth implements HasLogger { private static final String ATTR_AUTH = \u0026#34;authenticated\u0026#34;; private SessionAuth() { } public static boolean isAuthenticated() { VaadinSession session = VaadinSession.getCurrent(); if (session == null) return false; var attribute = session.getAttribute(ATTR_AUTH); staticLogger().info(\u0026#34;isAuthenticated.. {}\u0026#34;, attribute); return TRUE.equals(attribute); } public static void markAuthenticated() { staticLogger().info(\u0026#34;markAuthenticated.. \u0026#34;); VaadinSession session = VaadinSession.getCurrent(); if (session != null) { session.setAttribute(ATTR_AUTH, TRUE); } } public static void clearAuthentication() { staticLogger().info(\u0026#34;clearAuthentication.. \u0026#34;); VaadinSession session = VaadinSession.getCurrent(); if (session != null) { session.setAttribute(ATTR_AUTH, null); session.close(); } } } isAuthenticated() – Check if a session is logged in # The isAuthenticated() method is the primary read-only interface. It is used, among other things, in the MainLayout to decide whether navigation should be allowed or redirected to the login view as part of route protection. Internally, she first asks about the current VaadinSession. If no session exists (for example, in the very early stages of a request), it returns false immediately.\nIf a session exists, the \u0026ldquo;authenticated\u0026rdquo; attribute is read. It is a simple object value set to \u0026ldquo;Boolean.TRUE\u0026rdquo; upon successful login. The method compares this value to TRUE and returns true or false accordingly. The additional log entry helps track how often and with what results this check is invoked in the running system.\nmarkAuthenticated() – Mark session as logged in # After a successful password check in the LoginView, markAuthenticated() is called. The method fetches the current VaadinSession and sets the \u0026ldquo;authenticated\u0026quot; attribute to TRUE. This updates the session state so that subsequent calls to isAuthenticated() return true for that session.\nHere, too, a log entry ensures that the registration time remains traceable. If unexpected conditions occur in the company – for example, because users report that they are \u0026ldquo;suddenly logged out again\u0026rdquo; – the logs can be used to understand better when sessions were marked or discarded.\nclearAuthentication() – log out and close session # The third method, clearAuthentication(), is the counterpart to login. The logout button in the MainLayout invokes it and performs two tasks simultaneously. First, it removes the \u0026quot;authenticated\u0026quot; attribute from the current session by setting it to null. This means isAuthenticated() will return false for this session from this point on.\nIn the second step, \u0026ldquo;session.close()\u0026quot; is called. This invalidates the entire VaadinSession, including any other attributes that may have been set. This measure ensures that other session-related information is not forwarded and that a new request actually starts with a fresh session.\nIn combination with the explicit navigation back to the login page, this results in a clean logout flow: the session loses its authorisation, the user leaves the admin context, and must authenticate again to continue.\nInteraction with Login View and MainLayout # In the overall flow of the application, SessionAuth is used in several places:\nIn the LoginView, the attemptLogin()method callsSessionAuth.markAuthenticated()after a successful password comparisonand then navigates to the overview page. In the MainLayout, the beforeEnter() implementation uses SessionAuth.isAuthenticated() to intercept unauthorised access and, if necessary, redirect to the login view. The logout button in the header calls SessionAuth.clearAuthentication() and then redirects the user out of the administration context to the login page. In this way, SessionAuth centralises all access to the authentication status in a single place. Changes to the session model – such as a different attribute name or additional metadata – are made here and are then consistently available to all consuming components.\nConclusion \u0026amp; Outlook # With the introduction of a simple, configurable admin login, the application gains a clearly defined protection mechanism for all administrative functions without the complexity of a full-fledged authentication framework. The solution is deliberately tailored to the project\u0026rsquo;s characteristics: lightweight, comprehensible, and implemented using pure Java/Vaadin architecture. At the same time, it covers the most essential requirements for minimal access control – from password requests to route protection to a clean logout mechanism.\nThe overall system consists of a few clearly separated building blocks: a central configuration file, the LoginConfig for evaluating these values, a simple login view for user interaction, route protection in the MainLayout, and the SessionAuth class for managing session state. These components interlock like gears, creating a process that remains technically sound and intuitive for users.\nAt the same time, the implementation is deliberately extensible. The current approach stores the password in plain text in the configuration file and uses it, unchanged, as a byte sequence within the application. This is sufficient for development and internal scenarios, but it is clear that it is not suitable for higher security requirements. However, the architecture leaves room for the following points of expansion:\nPassword hashing: The stored passwords are hashed using a hash function such as SHA-256 or bcrypt, so they are not stored in plaintext on the server. Multiple users or roles: The structure could be expanded to support various administrators with individual passwords. Time-limited sessions: An automatic logout due to inactivity would further enhance security. Two-Factor Authentication (2FA): For more demanding environments, a second level of security – for example via TOTP – could be added. External identity providers: The application could be connected to OAuth 2.0/OpenID Connect in the long term, provided it fits the application scenario. However, the strength of the solution lies precisely in its not attempting to anticipate these aspects. It provides a pragmatic, ready-to-use foundation that reliably protects the admin area from unintended access without unnecessarily complicating deployment or development.\nCheers Sven\n","date":"18 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-basic-login-solution-part-2/","section":"Posts","summary":"What has happened so far? # In the first part of “Basic Login Solution”, the foundation for a deliberately simple yet structurally clean admin login was laid. The starting point was the realisation that even a technically lean URL shortener requires a clear separation between public-facing functions and administrative operations. The goal was not complete user management, but rather a minimal access barrier that integrates seamlessly with the existing Java and Vaadin architectures.\n","title":"Advent Calendar 2025 - Basic Login Solution - Part 2","type":"posts"},{"content":" Introduction # The administration interface of a URL shortener is a sensitive area where short links can be changed, removed, or assigned expiration dates. Although the system is often operated on an internal server or in a private environment, protecting this management interface remains a fundamental security concern. Accidental access by unauthorised users can not only lead to incorrect forwarding or data loss, but also undermine trust in the overall system.\nWith the introduction of a configurable login mechanism, precise access control is introduced, deliberately separating access to management functions from the rest of the system. The login serves as a lightweight security measure that does not require external dependencies, frameworks or time-consuming user management. It is precisely this simplicity – a single password, a simple configuration file and a centred login screen – that makes the solution particularly attractive for small deployments or personal projects.\nThe source code for this article can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-09\nObjectives of the implementation # The introduction of a simple, configurable login mechanism aims to achieve several clearly defined goals that improve both the security and usability of the URL shortener. The focus is not on establishing a complex user and role model, but on creating a pragmatic protective layer that reliably controls administrative access without unnecessarily increasing development or operating costs.\nFirst, the primary objective is to secure administrative functions. The management interface allows you to create, edit, and remove short links, as well as set or clean expiration dates. These functions must be protected against unauthorised access, particularly if the URL shortener is not operated exclusively within the closed network. A simple login prevents accidental or curious access from causing damage.\nAnother goal is to minimise the barrier to entry. The solution should work without additional frameworks, external identity providers, or complex configuration. The implementation is deliberately based on the project\u0026rsquo;s philosophy: Everything remains manageable, lightweight, and understandable. With a simple auth.properties file, the system can be turned on or off at will, and a single password suffices for access control.\nIn addition, the login mechanism is designed to ensure a predictable and consistent user experience. These include a well-designed login page, an immediate redirect on unauthenticated access to protected areas, and a transparent logout process that cleanly ends the session. All these aspects ensure that the login fits naturally into the existing UI and is still clearly perceived as a security measure.\nFinally, the login also serves as a foundation for potential future expansions. Although the current implementation is deliberately minimalist, it provides the structural prerequisites to respond to stronger security requirements if necessary—for example, by adding hashing mechanisms, time-limited session tokens, or server-side password rotation. This leaves the system open for further development without losing its current simplicity.\nConfigurable login via auth.properties # The new login mechanism is based on a deliberately simple configuration file that enables centralised control of the application\u0026rsquo;s authentication behaviour. This file is named auth.properties and is located in the application\u0026rsquo;s resource directory. By using them, the login system can not only be conveniently activated or deactivated, but also quickly adapted to different deployment scenarios.\nThe focus is on two configuration keys: login.enabled and login.password. While the first parameter checks whether the login system is active, the second parameter specifies the required access password. These values are automatically read when the application starts and henceforth determine the behaviour of all protected areas. It is precisely this mechanism that enables turning off login on short notice or changing the password without modifying the source code again.\nThe configuration is read in by the LoginConfigInitializer, which is executed when the servlet container starts. It checks whether the file exists, loads its contents and passes it to the central class LoginConfig, which manages these values and makes them available for later access. ``\nThis class is responsible for correctly configuring the application\u0026rsquo;s login behaviour at startup. This ensures that both the login page and the route protection can access a uniform configuration basis at all times.\nAfter loading the file, the LoginConfig class assumes responsibility for persistently storing the values and performing password comparisons.\nThis form of configuration control not only simplifies operation but also supports different operating modes. For example, a fully disabled login system is suitable for purely internal development environments, while the activated mode protects against unauthorised changes in production or publicly available environments. Switching between the two modes is kept as low-threshold as possible – a simple value in a text file is enough.\nAlthough this mechanism provides basic protection, it is not designed to be a highly secure authentication solution. Instead, the goal is to establish a minimum level of security to protect the administrative area from accidental access or unauthorised use. For productive environments with increased security requirements, additional measures such as password hashing, multi-factor authentication, or integration into established identity services would be necessary.\nLoginConfig \u0026amp; Initialization # Central control of login behaviour is based on two closely interlinked components: the LoginConfig class itself and its upstream initialiser, LoginConfigInitializer. Together, they ensure that the configuration from the auth.properties file is correctly read, interpreted, and made available to the application runtime.\nThe first focus is on the LoginConfig class. It provides a minimalist yet coherent foundation for the login system. The approach is intentionally simple: there is no user base, roles, or profiles, just a single password that serves as an access threshold. The class manages this password and the information on whether the login should be active. The structure remains manageable to keep the entry barrier for administrators and developers low.\nAn essential detail is the division into two phases: first, when the application starts, the LoginConfigInitializer checks which settings are stored in the configuration file. These values are then passed to the static LoginConfig.initialise() method, which populates the appropriate fields and makes them available to the rest of the application.\nThis initialisation process occurs entirely when the servlet container starts. This ensures that all views loaded later, especially the route protection and the login page, have access to a consistent and fully initialised configuration. This avoids error conditions that could arise from missing or delayed loading of configuration values.\nIn the following, we will take a closer look at both components and refer to the source code for clarity.\nLoginConfig – central configuration class # The core is the LoginConfig class, which stores the login switch and the expected password as byte data. It is declared final and has a private constructor, so it is used exclusively through its static methods:\n/** * Central configuration for the simple admin login. * Reads its values from auth.properties via LoginConfigInitializer. */ public final class LoginConfig { private static volatile boolean loginEnabled; private static volatile byte[] expectedPasswordBytes; private LoginConfig() { } /** * Initialises the login configuration. * * @param enabled whether the login mechanism should be enforced * @param password the raw password read from configuration, may be {@code null} */ public static void initialise(boolean enabled, String password) { loginEnabled = enabled; if (!enabled || password == null || password.isBlank()) { expectedPasswordBytes = null; return; } expectedPasswordBytes = password.getBytes(StandardCharsets.UTF_8); } /** * @return {@code true} if login protection is enabled at all */ public static boolean isLoginEnabled() { return loginEnabled; } /** * @return {@code true} if login is enabled and a usable password has been configured */ public static boolean isLoginConfigured() { return loginEnabled \u0026amp;\u0026amp;expectedPasswordBytes != null \u0026amp;\u0026amp; expectedPasswordBytes.length \u0026gt; 0; } /** * Compares the entered password with the configured one using constant-time comparison. */ public static boolean matches(char[] enteredPassword) { if (!isLoginConfigured() || enteredPassword == null) { return false; } byte[] entered = new String(enteredPassword).getBytes(StandardCharsets.UTF_8); boolean result = MessageDigest.isEqual(expectedPasswordBytes, entered); Best-effort clean-up Arrays.fill(entered, (byte) 0); return result; } } The initialise method is called only when the initialiser starts and determines whether login is enabled (loginEnabled) and whether a valid password is present. Invalid or empty passwords are consistently discarded; isLoginConfigured() only returns true if there is a usable configuration.\nFor the actual password comparison, matches(char[] enteredPassword) serves as the central function. It takes the entered password as a char[], converts it to a byte array in UTF-8 format, and compares it with the expected byte sequence using MessageDigest.isEqual. This enables constant-time comparison, making simple timing attacks more difficult. The temporary byte array is then overwritten using Arrays.fill to remove the sensitive data from memory, at least as effectively as possible.\nLoginConfigInitializer – Load configuration on startup # For LoginConfig to work with meaningful values, the configuration file must be read when the servlet container starts. This task is performed by the LoginConfigInitializer class, which is registered as @WebListener and implements the ServletContextListener interface :\n@WebListener public class LoginConfigInitializer implements ServletContextListener, HasLogger { private static final String PROPERTIES_PATH = \u0026#34;auth.properties\u0026#34;; @Override public void contextInitialized(ServletContextEvent sce) { logger().info(\u0026#34;Initialising LoginConfig from {}\u0026#34;, PROPERTIES_PATH); Properties props = new Properties(); try (InputStream in = getClass() .getClassLoader() .getResourceAsStream(PROPERTIES_PATH)) { if (in == null) { logger().warn(\u0026#34;No {} found on classpath. Login will be disabled.\u0026#34;, PROPERTIES_PATH); LoginConfig.initialise(false, null); return; } props.load(in); String enabledRaw = props.getProperty(\u0026#34;login.enabled\u0026#34;, \u0026#34;true\u0026#34;).trim(); boolean enabled = Boolean.parseBoolean(enabledRaw); String password = props.getProperty(\u0026#34;login.password\u0026#34;); LoginConfig.initialise(enabled, password); if (!enabled) { logger().info(\u0026#34;Login explicitly disabled via login.enabled=false\u0026#34;); } else if (LoginConfig.isLoginConfigured()) { logger().info(\u0026#34;LoginConfig initialised successfully from {}\u0026#34;, PROPERTIES_PATH); } else { logger().warn(\u0026#34;login.enabled=true but no usable password configured. \u0026#34; + \u0026#34;Login will effectively be disabled.\u0026#34;); } } catch (IOException e) { logger().error(\u0026#34;Failed to load \u0026#34; + PROPERTIES_PATH + \u0026#34;. Login will be disabled.\u0026#34;, e); LoginConfig.initialise(false, null); } } @Override public void contextDestroyed(ServletContextEvent sce) { Nothing to clean up } } The initialiser follows a straightforward process: first, it attempts to load the auth.properties file from the classpath. If this is not possible, the login is explicitly deactivated, and a warning log is written. If successful, the configuration values \u0026ldquo;login.enabled\u0026quot; and \u0026ldquo;login.password\u0026quot; are read, converted to appropriate data types, and forwarded to LoginConfig.initialise. Finally, depending on the configuration, additional log output is generated, providing quick information about the active mode during operation.\nThis separation of responsibilities creates a well-traceable initialisation path: LoginConfigInitializer handles loading and interpreting the configuration file. At the same time, LoginConfig encapsulates the actual login logic and is later used by other components, such as the login view or route protection.\nThe new login page # With the introduction of admin login, the application gains its own entry page that deliberately distinguishes itself from the rest of the UI. The new login page serves as the boundary between public functions, such as link forwarding, and sensitive administrative functions in the backend. The goal was to create a reduced, highly focused layout that integrates seamlessly with the application\u0026rsquo;s visual design while remaining clearly recognisable as a security barrier.\nInstead of being embedded in the existing navigation layout, the login is rendered as a standalone view without MainLayout. Users will not see any page navigation, drawer or admin functions until they have successfully authenticated. The page fills the entire browser window; its contents are centred vertically and horizontally. This creates the impression of a classic login screen with a single task: requesting the password for the administrative area.\nAt the centre is a compact login panel comprising a headline, a short explanatory text, a password field, and a login button. The headline clearly conveys that this is admin access, while the accompanying text explains why a password is required and what area it protects. This improves transparency and reduces queries, especially when multiple users use the system.\nThe password field is configured to automatically receive focus when the page is loaded. Users can therefore type directly without first clicking with the mouse. In addition, a clear button lets you remove an accidentally entered string with a single click. Displaying the password in plain text is deliberately avoided to reduce the risk of shoulder-surfing, especially in shared work environments.\nThe input behaviour is also designed to ensure a smooth process. Authentication can be done either by clicking on the login button or by pressing Enter. If the login fails, the view marks the password field as invalid and displays a clear error message indicating that the entered password is incorrect. This preserves context and allows the user to start a second attempt immediately without reloading the page.\nImplementation of the LoginView # The technical implementation of the described login page is handled by the LoginView class. It is registered as a separate route and deliberately dispenses with a layout to enable a focused, full-surface login screen:``\n@Route(LoginView.PATH) // No layout = no navigation or drawer visible @PageTitle(\u0026#34;Admin Login | URL Shortener\u0026#34;) public class LoginView extends VerticalLayout implements BeforeEnterObserver { public static final String PATH = \u0026#34;login\u0026#34;; private final PasswordField passwordField = new PasswordField(\u0026#34;Password\u0026#34;); private final Button loginButton = new Button(\u0026#34;Login\u0026#34;); public LoginView() { setSizeFull(); setAlignItems(Alignment.CENTER); setJustifyContentMode(JustifyContentMode.CENTER); configureForm(); buildLayout(); } It is already evident from the declaration that LoginView is not embedded in MainLayout. Because the route annotation lacks a layout, the page is displayed in isolation; the layout properties in the constructor handle complete vertical and horizontal centring. The two central UI components – password field and login button – are kept as fields so that they can be further configured in the helper methods configureForm()andbuildLayout().\nThe configuration of the form is deliberately designed for a lean but fluid user experience:\nprivate void configureForm() { passwordField.setAutofocus(true); passwordField.setWidth(\u0026#34;300px\u0026#34;); passwordField.setClearButtonVisible(true); passwordField.setRevealButtonVisible(false); passwordField.setInvalid(false); loginButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY); loginButton.setWidth(\u0026#34;300px\u0026#34;); loginButton.addClickListener(_ -\u0026gt; attemptLogin()); passwordField.addKeyDownListener(event -\u0026gt; { if (\u0026#34;Enter\u0026#34;.equalsIgnoreCase(event.getKey().getKeys().getFirst())) { attemptLogin(); } }); passwordField.addValueChangeListener(_ -\u0026gt; passwordField.setInvalid(false)); } The password field automatically gains focus when the page loads and is set to a fixed width, so that the input and button visually form a single unit. The Clear button lets you quickly delete an incorrect entry; the password is displayed in plain text by default. The login button is marked as the primary button and, when pressed, triggers the attemptLogin() method. The value change listener ensures that a previously set error state is restored when typing again.\nThe structure of the panel that appears in the middle of the screen is defined in buildLayout():\nprivate void buildLayout() { H2 title = new H2(\u0026#34;Admin Login\u0026#34;); Paragraph subtitle = new Paragraph( \u0026#34;Please enter the administrator password to access the management interface.\u0026#34; ); VerticalLayout formLayout = new VerticalLayout(title, subtitle, passwordField, loginButton); formLayout.setSpacing(true); formLayout.setPadding(true); formLayout.setAlignItems(Alignment.CENTER); add(formLayout); } The combination of the heading, explanatory paragraph, password field, and button forms exactly matches the compact login panel described in the previous section. Centring the inner VerticalLayout creates a clear focus on the input area, without visual distractions from other UI elements.\nThe actual login process is encapsulated in attemptLogin(). This is where configuration and user input are merged:\nprivate void attemptLogin() { if (! LoginConfig.isLoginEnabled()) { Notification.show( \u0026#34;Login is currently disabled. Please check the server configuration.\u0026#34; 3000, Notification.Position.MIDDLE ); UI.getCurrent().navigate(OverviewView.PATH); return; } char[] input = passwordField.getValue() != null ? passwordField.getValue().toCharArray() : new char[0]; if (! LoginConfig.isLoginConfigured()) { Notification.show( \u0026#34;Login is not configured. Please verify that the configuration file has been loaded.\u0026#34;, 3000, Notification.Position.MIDDLE ); return; } boolean authenticated = LoginConfig.matches(input); if (authenticated) { SessionAuth.markAuthenticated(); UI.getCurrent().navigate(OverviewView.PATH); } else { passwordField.setErrorMessage(\u0026#34;Incorrect password\u0026#34;); passwordField.setInvalid(true); } } First, it is checked whether the login is enabled in the configuration. If this is not the case, the user receives clear feedback via notification and is redirected directly to the overview page. The next step is to check whether the login has been configured correctly. Only when both conditions are met is the entered password passed to LoginConfig.matches. If successful, the session is marked as authenticated via SessionAuth.markAuthenticated() and forwarded to the overview. In the event of an error, the view marks the password field as invalid and displays a clear error message – the user remains on the login page and can adjust the input.\nFinally, the implementation of the BeforeEnterObserver ensures that the login page itself only remains visible when it makes sense:\n@Override public void beforeEnter(BeforeEnterEvent event) { If login is disabled, skip the login page entirely if (! LoginConfig.isLoginEnabled()) { event.forwardTo(OverviewView.PATH); return; } If already authenticated, also skip the login page if (SessionAuth.isAuthenticated()) { event.forwardTo(OverviewView.PATH); } } This prevents authenticated users from being redirected to the login page again and ensures This prevents authenticated users from being redirected to the login page again and ensures that a globally deactivated login system does not unnecessarily display a redundant password dialogue.\nAnother aspect is the interaction with the configuration. If login in the auth.properties file is disabled or not configured meaningfully, the view displays clear instructions. In one case, it indicates that login is currently disabled and redirects directly to the overview page. In the other case, it suggests that a valid configuration could not be loaded. The page thus serves not only as an input form, but also as a diagnostic point where misconfigurations become visible at an early stage.\nOverall, the new login page ensures that access to the admin interface is structured, predictable and visually clearly distinguished from the rest of the system. It combines a deliberately simple interaction with a comprehensible security concept, thereby laying the foundation for the other building blocks of the login flow, particularly route protection and session management.\nCheers Sven\n","date":"17 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-basic-login-solution-part-1/","section":"Posts","summary":"Introduction # The administration interface of a URL shortener is a sensitive area where short links can be changed, removed, or assigned expiration dates. Although the system is often operated on an internal server or in a private environment, protecting this management interface remains a fundamental security concern. Accidental access by unauthorised users can not only lead to incorrect forwarding or data loss, but also undermine trust in the overall system.\n","title":"Advent Calendar 2025 - Basic Login Solution - Part 1","type":"posts"},{"content":" What has happened so far\u0026hellip; # In the previous part, the URL shortener overview was significantly expanded. The starting point was the realisation that the last UI was heavily optimised for single operations and thus quickly reached its limits as soon as larger volumes of shortlinks needed to be managed. To resolve this bottleneck, the grid was consistently switched to multiple selection, creating the technical basis for actual mass operations.\nBased on this, a context-dependent bulk action bar was created, visible only when a selection is present. It serves as a central control for simultaneous multi-entry actions and seamlessly integrates them into the existing search, filtering, and grid interaction workflow. The decisive factor was not the mere addition of new buttons, but the clean coupling of UI state, selection and action permission.\nThe first concrete mass operation was the bulk delete. This function exemplified how efficiency and security can be combined: through explicit confirmation dialogues, meaningful previews of the affected entries, consistent feedback, and clear visual cues for destructive actions. This workflow has been supplemented with keyboard shortcuts that speed up deleting larger quantities without bypassing the control mechanisms.\nThis completes the transition from an entry-centric interface to a professional management view. The overview view now understands selections as a work context and can execute coordinated actions in a controlled manner. The following sections build on this foundation, examining further mass operations, their semantic differences, and their technical implementation in detail.\nThe source code for this article can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-08\nBulk Clear Expiry: Reliably and Consistently Remove Expiration Dates # While setting a uniform expiration date is a typical administrative task, the targeted removal of such expiration dates also plays a central role. In many real-world scenarios, temporary restrictions lose their relevance: links should remain valid permanently, test or campaign parameters must be reset, or previously set durations have proven too restrictive. The ability to delete expiration dates for many shortlinks in a single step is, therefore, a crucial building block for effective mass management.\nThe bulk clear expiry function follows a similar interaction pattern to setting an expiration date. Still, it is fundamentally different in its consistency and semantics: while setting introduces a new value, deleting an expiration date explicitly returns to the \u0026ldquo;does not automatically expire\u0026rdquo; state. The architecture must ensure that this state is clearly communicated and implemented correctly.\nThe removal of the expiration time must be understood explicitly, not as an implicit side effect, but as a deliberate operation that requires its own confirmation. This ensures both the security of user interactions and the traceability of changes.\nThe focus is on the interaction among dialogue logic, API calls, and the changed semantics of the edit endpoint, which now distinguishes between \u0026ldquo;set new expiration date\u0026rdquo; and \u0026ldquo;delete expiration date\u0026rdquo;. This makes the interaction between the UI and the backend clearly comprehensible and demonstrates how reliable mass changes can be integrated cleanly.\nThe entry point of the functionality lies in the method confirmBulkClearExpirySelected(), which – analogous to the bulk delete – first validates the current selection and then opens a confirmation dialogue:\nprivate void confirmBulkClearExpirySelected() { var selected = grid.getSelectedItems(); if (selected.isEmpty()) { Notification.show(\u0026#34;No entries selected\u0026#34;); return; } Dialog dialog = new Dialog(); dialog.setHeaderTitle(\u0026#34;Remove expiry for \u0026#34; + selected.size() + \u0026#34; short links?\u0026#34;); dialog.add(new Text( \u0026#34;This will remove the expiry date from all selected short links. \u0026#34; + \u0026#34;They will no longer expire automatically.\u0026#34; )); Button cancel = new Button(\u0026#34;Cancel\u0026#34;, _ -\u0026gt; dialog.close()); Button confirm = new Button(\u0026#34;Remove expiry\u0026#34;, _ -\u0026gt; { dialog.close(); bulkClearExpiry(selected); }); confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY); dialog.getFooter().add(new HorizontalLayout(cancel, confirm)); dialog.open(); } The structure of this method makes it clear that removing the expiration time is modelled as a standalone operation that requires confirmation. First, an empty selection is excluded; then a dialogue is created whose title explicitly names both the operation type (\u0026ldquo;Remove expiry\u0026rdquo;) and the number of affected entries. The actual content of the dialogue explains the consequence of the action in plain language: The links \u0026ldquo;will no longer expire automatically\u0026rdquo;. This means that semantics are conveyed not only implicitly by the UI but also explicitly by language.\nThe actual implementation of the mass operation is carried out in the outsourced method bulkClearExpiry(...), which encapsulates the transition from the UI layer to the client API:\nprivate void bulkClearExpiry(Set\u0026lt;ShortUrlMapping\u0026gt; selected) { if (selected.isEmpty()) { Notification.show(\u0026#34;No entries selected\u0026#34;); return; } int success = 0; int failed = 0; for (var m : selected) { try { boolean ok = urlShortenerClient.edit( m.shortCode(), m.originalUrl() ); if (ok) { success++; } else { failed++; } } catch (IOException ex) { logger().error(\u0026#34;Bulk clear expiry failed for {}\u0026#34;, m.shortCode(), ex); failed++; } } grid.deselectAll(); safeRefresh(); Notification.show(\u0026#34;Cleared expiry: \u0026#34; + success + \u0026#34; • Failed: \u0026#34; + failed); } The signature of the edit call is particularly remarkable. In contrast to the bulk set expiry, only the shortcode and original URL are transferred; the instant and shortcode are not. This API is explicitly implemented in the URLShortenerClient in such a way that the absence of an expiration date is semantically interpreted as \u0026ldquo;delete expiry\u0026rdquo;:\npublic boolean edit(String shortCode, String newUrl) throws IOException { return edit(shortCode, newUrl, null); } public boolean edit(String shortCode, String newUrl, Instant expiresAtOrNull) throws IOException { if (shortCode == null || shortCode.isBlank()) { throw new IllegalArgumentException(\u0026#34;shortCode must not be null/blank\u0026#34;); } if (newUrl == null || newUrl.isBlank()) { throw new IllegalArgumentException(\u0026#34;newUrl must not be null/blank\u0026#34;); } final URI uri = serverBaseAdmin.resolve(PATH_ADMIN_EDIT + \u0026#34;/\u0026#34; + shortCode); final URL url = uri.toURL(); logger().info(\u0026#34;edit - {}\u0026#34;, url); // ... Request set-up and dispatch ... } Finally, on the server side, this semantics is anchored in the store implementations. Both the in-memory store and the EclipseStore-based persistence layer now interpret a null value for expiredAt as a signal to remove the expiration time entirely:\nInMemoryUrlMappingStore var originalOrNewUrl = url != null ? url : shortUrlMappingOLD.originalUrl(); var instant = Optional.ofNullable(expiredAt); var shortUrlMapping = new ShortUrlMapping(shortCode, originalOrNewUrl, shortUrlMappingOLD.createdAt(), instant); store.put(shortUrlMapping.shortCode(), shortUrlMapping); EclipseUrlMappingStore var originalOrNewUrl = url != null ? url : shortUrlMappingOLD.originalUrl(); Optional\u0026lt;Instant\u0026gt; instant = Optional.ofNullable(expiredAt); var shortUrlMapping = new ShortUrlMapping(shortCode, originalOrNewUrl, shortUrlMappingOLD.createdAt(), instant); urlMappings.put(shortUrlMapping.shortCode(), shortUrlMapping); storage.store(dataRoot().shortUrlMappings()); Instead of replacing a missing value with the previous expiration date, expiredAt is now consistently converted to Optional.ofNullable(…). The result is a clear division of semantics: if an instant is passed, the edit operation sets a new expiration date; if null is passed, an empty optional is created, which corresponds to a complete removal of the existing expiration date. This change enables bulkClearExpiry(\u0026hellip;) not only to overwrite expiry information covertly but also to delete it.\nThe combination of a confirmation dialogue, dedicated mass operation, and precisely defined edit semantics creates a continuous path from the user click to the permanently changed data structure. Bulk-Clear-Expiry is not just a \u0026ldquo;special case\u0026rdquo; of the edit function, but a deliberately modelled, independent mass operation with clear, technically and professionally comprehensible meaning.\nKeyboard interaction as an accelerator for mass operations # With the introduction of mass grid operations, efficient interaction is becoming increasingly important. While mouse gestures and explicit buttons offer high visibility and explainability, keyboard shortcuts are most effective when users perform recurring actions. In the context of the Overview view, this means that central measurement operations should not only be accessible via the bulk bar but also be able to be triggered directly via the keyboard.\nKeyboard interaction fulfils two roles. On the one hand, it serves as a direct accelerator of already established workflows. If you are used to working with keyboard shortcuts, you can trigger searches and deletions without detours via the mouse, reducing noticeable friction, especially with frequent context switches between code, console, and browser. On the other hand, it serves as a semantic condensation of the UI: the presence of a shortcut makes it clear that certain operations are not treated as edge cases but as primary interaction paths.\nToday, this idea is being fleshed out in two prominent places. On the one hand, the global search in the overview view has a dedicated shortcut, allowing you to focus directly on the search field. This starts at the interaction\u0026rsquo;s beginning: a single keyboard shortcut is enough to switch from the web interface\u0026rsquo;s arbitrary state to a precisely defined search mode. On the other hand, the bulk delete operation is bound to the delete key only when a selection is present in the grid. The keyboard thus serves as an equal trigger for the same confirmation dialogue, which would otherwise be accessible only via the bulk bar.\nIt is essential for this integration that the keyboard shortcuts are not implemented as hidden, hard-to-discover functions, but as a consistent extension of the existing interaction model. They draw on concepts that have already been introduced – global search, multiple selection, bulk dialogues – and link them to a fluid, operable overall system. If you work exclusively with the mouse, you can use all functions as usual; those who prefer keyboard shortcuts, on the other hand, get much faster access to the same function space.\nIn the following, we will show how these shortcuts are technically integrated, how they interact with the grid state model, and which protective mechanisms prevent mass operations from being unintentionally triggered.\nThe central entry point is the addShortCuts() method, which bundles all global keyboard shortcuts of the Overview view:\nprivate void addShortCuts() { UI current = UI.getCurrent(); current.addShortcutListener(_ -\u0026gt; { if (globalSearch.isEnabled()) globalSearch.focus(); }, Key.KEY_K, KeyModifier.META); // Bulk delete via Delete-Key current.addShortcutListener(_ -\u0026gt; { if (!grid.getSelectedItems().isEmpty()) { confirmBulkDeleteSelected(); } }, Key.DELETE); } The method defines two keyboard interactions, each of which fulfils different semantic roles:\nMeta + K– Focus on the global search field # The combination of KeyModifier.META (⌘ on macOS, Windows logo key on Windows) and the letter K address an established UI pattern in code editors and command palettes. With a single keystroke, the user can go directly to the global search, regardless of which UI element was previously in focus.\nThe protection provided by the following conditions is remarkable:\nif (globalSearch.isEnabled()) globalSearch.focus(); This prevents the shortcut from forcing an interaction that would not make sense semantically in the current UI state – for example, if a dialogue is currently open or the search has been deactivated due to active filter modes. The keyboard interaction thus fits respectfully into the overall system.\nDelete– Trigger bulk delete if there is a selection # The second central key combination binds the delete key directly to the bulk delete function. The shortcut applies to the entire overview view, but is only activated if there is actually a multiple or single selection:\nif (!grid.getSelectedItems().isEmpty()) { confirmBulkDeleteSelected(); } This ensures that the delete dialogue appears only when there is an object context; therefore, it is not possible to trigger an operation without a target by randomly pressing Delete. This protection mechanism is essential for mass operations, as the delete key is often used unconsciously and intuitively.\nIntegration into the UI‑state model # Both shortcuts interfere with the grid\u0026rsquo;s state logic. The search shortcut uses the activation logic of the globalSearch field, while the delete shortcut is explicitly bound to the grid selection. This creates a natural coupling between keyboard interaction and UI state: Shortcuts are not detached from the UI, but follow the same semantic structure as mouse interaction via the bulk bar.\nWhy it matters: Acceleration without side effects # While shortcuts are often seen as optional conveniences, they play a crucial role in mass operations. Those who manage large amounts of data frequently switch between different tools – IDE, browser, and terminal. The ability to perform central operations without loss of focus and without visual navigation reduces cognitive load and increases speed. At the same time, the built-in protection queries ensure that no unintentional deletion actions are triggered.\nThis shows that keyboard interaction is not just a nice extra feature, but an integral part of an efficient, error-robust workflow for mass grid operations, how these shortcuts are integrated into the overview view, how they interact with the state model of the grid, and where protection mechanisms have been deliberately implemented to prevent unintentional mass operations.\nA consistent visual language for high-risk mass operations # With the introduction of mass operations, both the functional and visual levels become more complex. Where previously individual actions were placed in isolation in the grid, there are now interconnected chains of interactions that have the potential to have a significant impact on the database – especially in operations such as deleting several shortlinks or changing all expiration dates. This increased impact requires a visual language that creates clarity, clearly highlights risks and helps users to perform each operation consciously and safely.\nAn essential goal of this visual language is to establish clear semantic distinctions among action types. While harmless operations – such as opening a detailed dialogue – may be unagitated and neutral, risky or destructive actions must be immediately recognisable as such. The introduction of the bulk action bar makes this differentiation imperative: it bundles both low-risk and high-risk actions in a small space, creating visual clarity, a prerequisite for safe usability.\nThe implementation of these principles follows a strict pattern: by targeting Lumo theme variants, buttons are not only styled but also semantically enhanced. Destructive actions are consistently marked with LUMO_ERROR, whereas accompanying or secondary actions are given the more subtle LUMO_TERTIARY_INLINE. This contrast creates an immediately apparent hierarchy, highlighting potentially dangerous actions without visually cluttering the interface.\nThe technical basis of this visual language is first evident in the implementation of the bulk action bar. A clear distinction is made here between destructive and non-destructive actions. The corresponding code from the OverviewView clearly shows this:\nbulkDeleteBtn.addThemeVariants(LUMO_ERROR, LUMO_TERTIARY_INLINE); bulkSetExpiryBtn.addThemeVariants(LUMO_TERTIARY_INLINE); bulkClearExpiryBtn.addThemeVariants(LUMO_TERTIARY_INLINE); Only the Delete button bears the LUMO_ERROR mark. This signals at a glance that the action could have irreversible effects. The other two buttons – \u0026ldquo;Set expiry\u0026rdquo; and \u0026ldquo;Clear expiry\u0026rdquo; – deliberately remain inconspicuous. Their functions are operational, but not destructive, which is why LUMO_TERTIARY_INLINE offers the appropriate visual restraint here.\nThis differentiation also continues at the row level in the grid. Each row has two buttons that play completely different roles visually:\nButton delete = new Button(new Icon(VaadinIcon.TRASH)); delete.addThemeVariants(LUMO_ERROR, ButtonVariant.LUMO_TERTIARY); The delete button is red, making it immediately recognisable. The corresponding detail button is visually restrained:\nvar details = new Button(new Icon(VaadinIcon.SEARCH)); details.addThemeVariants(LUMO_TERTIARY_INLINE); The user immediately recognises which of the two actions is inconsequential and which is critical. This visual semantics is not cosmetic; it reduces misclicks and improves security, especially in fast workflows. This logic continues in the confirmation dialogues. In the case of bulk delete, the confirmation button is marked twice:\nButton confirm = new Button(\u0026#34;Delete\u0026#34;, _ -\u0026gt; { ... }); confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY, LUMO_ERROR); LUMO_PRIMARY marks the main action in the dialogue, LUMO_ERROR immediately makes it clear that this main action is destructive. The red colouring was deliberately chosen to increase attention and prevent accidental confirmations.\nHowever, different semantics apply when removing an expiration date. The process is a mass operation, but not destructive. Therefore, the application explicitly does not use a red marking:\nButton confirm = new Button(\u0026#34;Remove expiry\u0026#34;, _ -\u0026gt; { ... }); confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY); Here, only the primary theme is set – the action is essential, but not dangerous. The UI thus distinguishes between critical and relevant, which is crucial for intuitive usability.\nIn summary, the consistent use of Lumo themes creates a clear, recognisable hierarchy:\n**LUMO_ERROR** → destructive, potentially irreversible **LUMO_PRIMARY** → central affirmative action **LUMO_TERTIARY**/**LUMO_TERTIARY_INLINE** → harmless or accompanying actions This visual grammar helps users recognise the seriousness of an action without reading the text. It is precisely this clarity that is indispensable when operations are applied to many data sets rather than a single one.\nConclusion: From individual case to professional mass editing # With today\u0026rsquo;s update, the URL shortener\u0026rsquo;s overview view reaches a new level of functionality. What was previously a tool for individual case management is now developing into a full-fledged interface for professional mass editing. The fundamental expansion consists not only in the introduction of several bulk operations, but above all in how these operations are embedded within a consistent operating concept.\nIn the previous chapters, it became apparent that the introduction of multiple selection, bulk bar, dialogue mechanisms and a finely graded visual language is not a loose juxtaposition of individual features. Instead, all elements are intertwined: the search structures the data space, the selection defines the area of action, the bulk bar makes the mass operations visible, and the dialogues ensure conscious, comprehensible decisions. Finally, visual semantics provide orientation and support safe operation even in fast or routine workflows.\nThis combination of architecture and interaction design provides the foundation for functions that can be added in future expansion stages, including automated workflows, grouped changes, role-based sharing, and even domain-specific mass transformations. These changes thus not only provide a feature package but also the foundation for a productive, reliable, and extensible mass-operations architecture.\nCheers Sven\n","date":"16 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-mass-grid-operations-part-2/","section":"Posts","summary":"What has happened so far… # In the previous part, the URL shortener overview was significantly expanded. The starting point was the realisation that the last UI was heavily optimised for single operations and thus quickly reached its limits as soon as larger volumes of shortlinks needed to be managed. To resolve this bottleneck, the grid was consistently switched to multiple selection, creating the technical basis for actual mass operations.\n","title":"Advent Calendar 2025 - Mass Grid Operations - Part 2","type":"posts"},{"content":"The next stage of the Advent calendar focuses on further development, which becomes immediately noticeable once more shortlinks are used. The previous interaction patterns in the Overview view were geared toward individual operations and yielded only limited efficiency gains when processing many objects simultaneously. Today, this paradigm is being deliberately broken up and replaced by an interplay of new UI concepts that, for the first time, enable true multi-operation, context-sensitive work, and a much more stringent user interface.\nThe source code for this article can be found on GitHub at the following URL:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-08\nThe changes focus on three aspects: first, the introduction of a valid multiple selection, which was previously missing and now forms the basis for all new mass operations. Second, the contextual action bar, which appears only when relevant and blends seamlessly into the workflow. Third, a visibly improved depth of interaction, which is reflected in consistent visual cues, reduced refresh cycles, and the ability to perform more complex processing steps – such as setting or removing expiration dates – for any number of entries at the same time.\nThese improvements should not be viewed as a collection of isolated features, but rather as a coordinated step toward a more mature UI architecture. They address a common issue in production environments: users manage not just one but dozens or hundreds of shortlinks. Any optimisation that reduces friction losses here directly affects work speed and quality. This is exactly where this step has its impact: the application is not only faster to use but also more tailored to professional use cases.\nMultiple selection as the foundation for mass grid operations # The transition from a single to a multiple selection marks a fundamental change in the interaction logic of the Overview view. Until now, working with shortlinks has been limited to isolated operations that always followed the same sequence: an entry was selected, edited, confirmed, and the process started over again. This structure was functional but proved to be a stumbling block when handling a larger number of entries in real-world applications.\nWith the introduction of multiple selection, this bottleneck will be resolved. The grid transforms from a linear list into a workspace where various objects can be brought into focus simultaneously. This capability is not only an ergonomic improvement, but also forms the technical and conceptual foundation for all subsequent mass operations. Only the possibility of marking any number of shortlinks at the same time makes context-sensitive work possible at all.\nThe technical anchoring of this multiple selection is initially done at a very inconspicuous point in the grid configuration. The overview view explicitly switches from single to multi-selection mode, which means that the grid is internally able to keep several entries as selected in parallel:\nprivate void configureGrid() { logger().info(\u0026#34;configureGrid..\u0026#34;); grid.addThemeVariants(GridVariant.LUMO_ROW_STRIPES, GridVariant.LUMO_COMPACT); grid.setHeight(\u0026#34;70vh\u0026#34;); grid.setSelectionMode(Grid.SelectionMode.MULTI); With this change, the Vaadin grid\u0026rsquo;s selection mechanism will be expanded from the ground up. While in single-selection mode, the focus is technically always on exactly one entry in Grid.SelectionMode.MULTI allows multiple selections simultaneously. The basic interaction pattern mustn’t change for the user: the choice is still made via familiar gestures, such as clicks, supplemented by keyboard input as needed. The difference lies in the underlying representation: the grid now manages many marked shortlinks rather than a single active element.\nTo ensure that this multiple selection is not only visually effective, but also functionally reflected, it is integrated into the rest of the interface via a selection listener:\ngrid.addSelectionListener(event -\u0026gt; { var all = event.getAllSelectedItems(); boolean hasSelection = !all.isEmpty(); bulkBar.setVisible(hasSelection); if (hasSelection) { int count = all.size(); String label = count == 1 ? \u0026#34;link selected\u0026#34;: \u0026#34;links selected\u0026#34;; selectionInfo.setText(count + \u0026#34; \u0026#34; + label + \u0026#34; on page \u0026#34; + currentPage); } else { selectionInfo.setText(\u0026#34;\u0026#34;); } bulkDeleteBtn.setEnabled(hasSelection); bulkSetExpiryBtn.setEnabled(hasSelection); bulkClearExpiryBtn.setEnabled(hasSelection); }); Here, the abstract multiple selection is converted into a concrete UI state. The getAllSelectedItems() method returns the quantity of shortlinks currently marked in the grid. From this, a boolean aggregate state hasSelection is derived, which controls whether the downstream bulk bar is visible and whether the associated action buttons are enabled. At the same time, the selection size provides immediate feedback to the user: the number and context of the selection are summarised in a text component that makes the current page and the number of marked links visible. In this way, multiple-choice is not only managed internally but also translated into a clearly traceable, state-based interaction.\nOf particular relevance is the fact that multiple selection has been seamlessly integrated into the existing UI model. The choice is made via established patterns, such as checkbox clicks or keyboard shortcuts, so users can work intuitively and productively immediately without adapting. At the same time, the visual feedback remains precise: selected rows are clearly highlighted, and changes in the selection immediately update the subsequent contextual UI components.\nThus, multiple selection provides the necessary foundation for bulk operations, such as deleting, setting or removing expiration times, and other future functions, to be implemented securely and consistently. It is the architectural pivot point for transitioning a purely entry-oriented tool into a scalable management tool for professional use cases.\nThe bulk action bar is a context-sensitive control centre. # With the introduction of multiple selection, you can, for the first time, edit multiple short links at once. However, this capability alone is insufficient to establish a consistent, high-ergonomics mass-machining process. Only through the bulk action bar, which dynamically appears or disappears based on the current selection, does the overview view become a true hub for mass operations.\nThe central idea of this bar is that tools become visible only when they are semantically relevant. As long as no entry is selected, the interface remains tidy and slim. However, as soon as one or more rows have been selected, the function space opens for all operations that refer to a set of objects. This dynamic follows the principle of context-sensitive compression: the UI does not present all possible actions simultaneously, but only those that are meaningful and feasible at the current task.\nIn addition, the bulk bar is not only a visual addition but also has a clearly structured internal logic, which is already reflected at the field level of the view. In the overview view, the essential UI building blocks for mass operations are explicitly declared as separate components:\nprivate final Button bulkDeleteBtn = new Button(\u0026#34;Delete selected links...\u0026#34;); private final Button bulkSetExpiryBtn = new Button(\u0026#34;Set expiry for selected...\u0026#34;); private final Button bulkClearExpiryBtn = new Button(\u0026#34;Clear expiry for selected\u0026#34;); private final Span selectionInfo = new Span(); private final HorizontalLayout bulkBar = new HorizontalLayout(); This structure makes it clear that the bulk bar is treated as an independent UI element with its own buttons and a separate information component. The actual integration takes place immediately after the search bar, so that the mass actions are visually and functionally in proximity to global navigation and filtering:\npublic OverviewView() { add(new H2(\u0026#34;URL Shortener – Overview\u0026#34;)); initDataProvider(); add(buildSearchBar()); add(buildBulkBar()); configureGrid(); addListeners(); addShortCuts(); } In this way, a vertical structure is created in which search, bulk bar, and grid deliberately follow one another: First, the data space is filtered; then the selected elements are contextualised via the bulk bar before the detailed interaction takes place in the grid.\nThe behaviour of the bulk bar itself is encapsulated in a dedicated factory method that defines both the visual appearance and the initial functional state:\nprivate Component buildBulkBar() { bulkDeleteBtn.addThemeVariants(LUMO_ERROR, LUMO_TERTIARY_INLINE); bulkSetExpiryBtn.addThemeVariants(LUMO_TERTIARY_INLINE); bulkClearExpiryBtn.addThemeVariants(LUMO_TERTIARY_INLINE); bulkDeleteBtn.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Delete all selected short links\u0026#34;); bulkSetExpiryBtn.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Set the same expiry for all selected links\u0026#34;); bulkClearExpiryBtn.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Remove expiry for all selected links\u0026#34;); bulkDeleteBtn.setEnabled(false); bulkSetExpiryBtn.setEnabled(false); bulkClearExpiryBtn.setEnabled(false); bulkBar.removeAll(); selectionInfo.getStyle().set(\u0026#34;opacity\u0026#34;, \u0026#34;0.7\u0026#34;); selectionInfo.getStyle().set(\u0026#34;font-size\u0026#34;, \u0026#34;var(--lumo-font-size-s)\u0026#34;); selectionInfo.getStyle().set(\u0026#34;margin-right\u0026#34;, \u0026#34;var(--lumo-space-m)\u0026#34;); bulkBar.add(selectionInfo, bulkDeleteBtn, bulkSetExpiryBtn, bulkClearExpiryBtn); bulkBar.setWidthFull(); bulkBar.setDefaultVerticalComponentAlignment(Alignment.CENTER); bulkBar.setVisible(false); return bulkBar; } In this method, the semantic roles of the individual elements are concretised. The theme variants particularly highlight the delete button, which is provided with an error theme and thus clearly distinguishes itself from less risky operations. The tooltips also enhance the explainability of actions by precisely describing each button\u0026rsquo;s purpose. All buttons start in a deactivated state; only the selection in the grid – as shown in the previous chapter – unlocks it in a targeted manner.\nThe layout definition makes it clear that the bulk bar is intended to be a full-width, horizontally aligned control centre. The selectionInfo area is slightly smaller but serves as a permanent context anchor that shows the user which mass scenario they are currently in. The visibility of the entire bar is initially set to false and controlled solely by the grid\u0026rsquo;s selection state. This ensures the bulk bar is visible only when it is needed. It first provides precise information on the selection status by displaying the number of selected shortlinks and the current page. Subsequently, the buttons that can be executed on this basis are unlocked. This mechanism prevents operational errors and ensures the user immediately recognises which actions can currently be performed.\nBulk Delete: Secure and scalable deletion operations in the grid # With the transition to mass operations, deleting multiple shortlinks simultaneously is a key use case. The previous UI was clearly designed for deleting individual entries, which was insufficient for many professional use cases. Especially when it comes to administrative activities, migrations, or cleaning test data, there is a natural need to remove many entries in a single step. This is precisely where the new bulk delete function comes in.\nThe key requirement for this function is to balance two aspects: on the one hand, maximum efficiency when handling large volumes of entries, and on the other, high robustness against unintentional or incorrect entries. The system should work quickly, but never delete data carelessly. The implementation, therefore, follows a two-stage interaction pattern, deliberately designed to enable mass processing while ensuring the necessary operational safety.\nThe bulk delete always starts with the multiple selection in the grid. Once many shortlinks have been marked, the bulk action bar is activated, and the \u0026ldquo;Delete selected links\u0026hellip;\u0026rdquo; option becomes available. Only when this action is deliberately chosen does a separate confirmation dialogue open. It does more than ask a simple query; it displays sample shortcodes of the selected entries, making the deletion process transparent and verifiable for the user. This preview is essential for larger datasets, as it helps validate the selection\u0026rsquo;s accuracy without reading the entire list of selected items.\nThe confirmation pattern is designed to trigger the deletion process in a targeted manner without interrupting the workflow. Once confirmed, all highlighted shortlinks will be deleted one by one via the client API. The architecture supports both parallel and sequential deletion patterns; the current implementation uses a sequential approach to capture errors and report them to the UI precisely. Each deletion operation is evaluated, and the user receives a summary of the results that clearly documents both successful and failed deletions.\nThis combination of efficiency, security and transparency makes bulk delete a robust tool in a professional context. In the following, the relevant source texts are used to concretise and explain in detail how dialogue logic, API integration and feedback mechanisms interlock to realise a precisely controlled deletion process.\nThe entry point for bulk deletion is the confirmBulkDeleteSelected() method in the Overview view. It integrates the grid\u0026rsquo;s current multi-selection with the dialogue logic. It prepares both the preview of the entries to be deleted and the subsequent execution of the delete operations:\nprivate void confirmBulkDeleteSelected() { var selected = grid.getSelectedItems(); if (selected.isEmpty()) { Notification.show(\u0026#34;No entries selected\u0026#34;); return; } Dialog dialog = new Dialog(); dialog.setHeaderTitle(\u0026#34;Delete \u0026#34; + selected.size() + \u0026#34; short links?\u0026#34;); var exampleCodes = selected.stream() .map(ShortUrlMapping::shortCode) .sorted() .limit(5) .toList(); if (!exampleCodes.isEmpty()) { String preview = String.join(\u0026#34;, \u0026#34;, exampleCodes); if (selected.size() \u0026gt; 5) { preview += \u0026#34;, ...\u0026#34;; } dialog.add(new Text(\u0026#34;Examples: \u0026#34; + preview)); } else { dialog.add(new Text(\u0026#34;Delete selected short links?\u0026#34;)); } Button confirm = new Button(\u0026#34;Delete\u0026#34;, _ -\u0026gt; { int success = 0; int failed = 0; for (var m : selected) { try { boolean ok = urlShortenerClient.delete(m.shortCode()); if (ok) { success++; } else { failed++; } } catch (IOException ex) { logger().error(\u0026#34;Bulk delete failed for {}\u0026#34;, m.shortCode(), ex); failed++; } } dialog.close(); grid.deselectAll(); safeRefresh(); Notification.show(\u0026#34;Deleted: \u0026#34; + success + \u0026#34; • Failed: \u0026#34; + failed); }); confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY, LUMO_ERROR); Button cancel = new Button(\u0026#34;Cancel\u0026#34;, _ -\u0026gt; dialog.close()); dialog.getFooter().add(new HorizontalLayout(confirm, cancel)); dialog.open(); } The first block already clarifies the implementation\u0026rsquo;s security concept. If, contrary to expectations, no selection is made, the process will be aborted with an explicit notification. The actual dialogue instance receives a header that explicitly shows the number of affected shortlinks, thereby clarifying the scope of the action.\nParticularly interesting is how the preview of the entries to be deleted is generated. From the number of selected shortlinks, a sorted list of your shortcodes is created, which is limited to a maximum of five examples. This limitation ensures that the dialogue remains readable even with huge selections, while an appended ellipsis makes it clear that other elements are affected. This creates a concise but meaningful presentation of the selection.\nFinally, the inner Confirm handler performs the actual deletion process. For each shortlink selected, the client API of the URL shortener backend is called via urlShortenerClient.delete(m.shortCode()). Successful and failed operations are counted; Exceptions are explicitly logged and taken into account as errors in the count. This count serves as the basis for the final user notification, which summarises the number of successfully deleted and failed entries.\nAfter the loop completes, the dialogue is closed, the grid selection is cancelled, and a consistent update of the displayed data is triggered via safeRefresh(). The final notification provides precise, aggregated feedback on the operation outcome, thereby completing the interaction cycle of selection, confirmation, and presentation of results.\nIn addition, the bulk delete function can also be operated via the keyboard. The user can use a global shortcut to trigger the delete dialogue directly, provided that a selection has been made:\nprivate void addShortCuts() { UI current = UI.getCurrent(); current.addShortcutListener(_ -\u0026gt; { if (globalSearch.isEnabled()) globalSearch.focus(); }, Key.KEY_K, KeyModifier.META); current.addShortcutListener(_ -\u0026gt; { if (!grid.getSelectedItems().isEmpty()) { confirmBulkDeleteSelected(); } }, Key.DELETE); } This integrates the bulk delete into the everyday workflow: pressing the Delete key triggers the same confirmation dialogue as clicking the corresponding button in the bulk bar. Mouse and keyboard interaction thus yields the same result, significantly speeding up operations for experienced users without compromising the security of the operation.\nBulk Set Expiry: Uniform expiration dates for multiple shortlinks # With the possibility of selecting several entries at the same time, it is possible for the first time to make even more complex metadata changes in a single step. One of the most functionally essential operations in this context is setting uniform expiration dates for multiple shortlinks. This feature is particularly relevant in administrative scenarios, such as when a large number of temporary links are set to expire simultaneously or when test or campaign data needs to be centrally controlled.\nThe design of the bulk set expiry function is intended to be both precise and efficient. Users should be able to set the date and time in a consistent dialogue without having to navigate through each entry individually. At the same time, the implementation must robustly handle erroneous inputs and provide clear feedback on the operation\u0026rsquo;s success or failure.\nThe dialogue-based setting of a common expiration date follows a three-step interaction pattern. First, the user selects all relevant shortlinks, which makes the bulk bar visible and enables the \u0026ldquo;Set expiry for selected\u0026hellip;\u0026rdquo; option. In the second step, a custom dialogue opens to enter the date and time. Only in the third step, after the conscious confirmation, are the new expiration dates set for all selected entries and the grid is updated accordingly.\nThe entry point into this functionality is the openBulkSetExpiryDialog() method, which is triggered directly from the bulk action bar. It encapsulates both the dialogue-based recording of the expiration time and the later forwarding to the server API:\nprivate void openBulkSetExpiryDialog() { var selected = grid.getSelectedItems(); if (selected.isEmpty()) { Notification.show(\u0026#34;No entries selected\u0026#34;); return; } Dialog dialog = new Dialog(); dialog.setHeaderTitle(\u0026#34;Set expiry for \u0026#34; + selected.size() + \u0026#34; short links\u0026#34;); DatePicker date = new DatePicker(\u0026#34;Date\u0026#34;); TimePicker time = new TimePicker(\u0026#34;Time\u0026#34;); time.setStep(Duration.ofMinutes(15)); HorizontalLayout body = new HorizontalLayout(date, time); body.setAlignItems(Alignment.END); dialog.add(body); Button cancel = new Button(\u0026#34;Cancel\u0026#34;, _ -\u0026gt; dialog.close()); Button apply = new Button(\u0026#34;Apply\u0026#34;, _ -\u0026gt; { if (date.getValue() == null) { Notification.show(\u0026#34;Please select a date\u0026#34;); return; } var localTime = Optional.ofNullable(time.getValue()).orElse(LocalTime.of(0, 0)); var zdt = ZonedDateTime.of(date.getValue(), localTime, ZoneId.systemDefault()); Instant expiresAt = zdt.toInstant(); int success = 0; int failed = 0; for (ShortUrlMapping m : selected) { try { boolean ok = urlShortenerClient.edit( m.shortCode(), m.originalUrl(), expiresAt ); if (ok) success++; else failed++; } catch (IOException ex) { logger().error(\u0026#34;Bulk set expiry failed for {}\u0026#34;, m.shortCode(), ex); failed++; } } dialog.close(); grid.deselectAll(); safeRefresh(); Notification.show(\u0026#34;Updated expiry: \u0026#34; + success + \u0026#34; • Failed: \u0026#34; + failed); }); apply.addThemeVariants(ButtonVariant.LUMO_PRIMARY); dialog.getFooter().add(new HorizontalLayout(cancel, apply)); dialog.open(); } The method header already includes a security prompt that prevents the dialogue from opening accidentally without a valid selection. The dialogue configuration is deliberately kept minimalist and focuses on the core input elements: a DatePicker for the date and a TimePicker for the time. The TimePicker\u0026rsquo;s step size is set to 15 minutes, which is a practical compromise between precision and usability.\nThe core logic begins with validating the date. If no date is selected, the execution is cancelled, and a corresponding notification is issued. If a date is specified, the method creates a ZonedDateTime from it, which is then converted to an instant. This instant represents the new expiration time to be assigned to all selected shortlinks.\nThe following loop block illustrates the interface to the backend. For each shortlink, the edit operation is called:\nurlShortenerClient.edit(m.shortCode(), m.originalUrl(), expiresAt); This operation is designed to change only the expiration time, while preserving the original URL value. Errors are handled individually and accounted for in both the UI and logging. Summing up the successful and failed updates results in aggregated feedback at the end of the process, which immediately tells the user how successful the bulk operation was.\nFinally, grid.deselectAll() combined with safeRefresh() ensures that both the UI state and the grid data are updated consistently. The user is thus placed in a clean reset state without any manual intermediate steps, while the final Notification.show(\u0026hellip;) summarises the operation results. The bulk set expiry function illustrates the interplay among the UI, server API, and error handling, and explains how this mechanism greatly simplifies managing large numbers of links.\nCheers Sven\n","date":"15 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-mass-grid-operations-part-1/","section":"Posts","summary":"The next stage of the Advent calendar focuses on further development, which becomes immediately noticeable once more shortlinks are used. The previous interaction patterns in the Overview view were geared toward individual operations and yielded only limited efficiency gains when processing many objects simultaneously. Today, this paradigm is being deliberately broken up and replaced by an interplay of new UI concepts that, for the first time, enable true multi-operation, context-sensitive work, and a much more stringent user interface.\n","title":"Advent Calendar 2025 - Mass Grid Operations - Part 1","type":"posts"},{"content":"After the first part explained the conceptual basics and the new interactions among global search, search scopes, and advanced filters, the second part focuses on the technical mechanisms that enable these interactions. It is only the revised refresh architecture – above all the interaction of safeRefresh() and RefreshGuard – that ensures that the OverviewView remains calm, deterministic and predictable despite numerous potential triggers.\nSo while Part 1 describes what has changed in the user interface and why this structuring was necessary, Part 2 now shows in detail how the internal state machine works, how competing UI events are coordinated and why the View achieves the desired robustness in the first place.\nWith this foundation, the following chapters can be assigned clearly: they analyse the refresh architecture, the reset mechanism, and the validation and error-handling logic – the technical building blocks that ensure the UI concepts described above not only look good but also function reliably.\nThe source code for this development step can be found on GitHub\nand can be found here:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-07\nHere are some screenshots of the current development state.\nRefresh architecture with RefreshGuard # With the introduction of advanced filtering logic, not only does the complexity of the user interface increase, but so does the number of potential conditions that can cause a grid reload. Any change to a filter field, any adjustment to the page size, the opening or closing of the Advanced area, or entering the view itself can trigger a new request to the server. Without additional control, this could trigger a cascade of overlapping refreshes, create unnecessary load, and noticeably degrade the user experience.\nAgainst this background, the refresh architecture of the OverviewView has been specifically revised. The central goal was to reduce the number of possible triggers to a controllable mechanism that exhibits consistent, predictable behaviour across all relevant scenarios. Instead of allowing each UI event to have its own access to the DataProvider, a deliberate hard cut was made: the decision of whether and when a refresh may actually take place is bundled in a dedicated layer. This layer contains the RefreshGuard, which acts as a guardian that determines which processes are allowed to retrieve specific data and which are only allowed to change the internal state.\nThe technical basis of this control system is surprisingly simple at first. A single flag is introduced at the head of the class that controls whether refreshes are currently allowed:\nprivate boolean suppressRefresh = false; This flag is evaluated by a small helper method that acts as the only allowed entry point for data updates:\nprivate void safeRefresh() { logger().info(\u0026#34;safeRefresh\u0026#34;); if (!suppressRefresh) { logger().info(\u0026#34;refresh\u0026#34;); dataProvider.refreshAll(); } } Instead of reloading the grid or DataProvider from multiple locations, the view periodically calls safeRefresh(). The method first logs the refresh attempt, then checks the flag. Only if suppressRefresh is not set, refreshAll() is actually executed on the DataProvider. This creates a clear separation between the semantic intent to update the content and the operational decision of whether it is allowed in the current context.\nConceptually, the RefreshGuard can be understood as a critical section that allows multiple UI operations to be combined. As long as the code is within such a protected phase, direct refresh calls are suppressed. Only at the end of the section does the guard decide whether an aggregated refresh is required and, if so, execute it in a controlled manner. In the source code, this mechanism is implemented as a small, inner helper class:\nclass RefreshGuard implements AutoCloseable { private final boolean prev; private final boolean refreshAfter; RefreshGuard(boolean refreshAfter) { this.prev = suppressRefresh; this.refreshAfter = refreshAfter; suppressRefresh = true; } @Override public void close() { suppressRefresh = prev; if (refreshAfter) safeRefresh(); } } The constructor of the guard first stores the current state of suppressRefresh and then sets the flag to true. This means that all safeRefresh() operations called within this block are ineffective; they are logged but do not trigger a reload. When leaving the block (i.e., in the close() method), the previous flag value is restored. Optionally, a final refresh can then be triggered by the refreshAfter parameter. By implementing AutoCloseable, the guard can be used in a try-with-resources block, ensuring that the state is reset correctly even in exceptional cases.\nThe use of this pattern is particularly evident in the View lifecycle. When attaching the OverviewView to the UI, the column visibilities are first set based on the stored preferences. This process changes the grid state significantly but should not cause new data to be loaded from the server multiple times. Accordingly, the entire block is wrapped in a RefreshGuard :\n@Override protected void onAttach(AttachEvent attachEvent) { super.onAttach(attachEvent); try (var _ = new RefreshGuard(true)) { var keys = grid .getColumns() .stream() .map(Grid.Column::getKey) .filter(Objects::nonNull) .toList(); var vis = columnVisibilityService.mergeWithDefaults(keys); grid.getColumns().forEach(c -\u0026gt; { var k = c.getKey(); if (k != null) c.setVisible(vis.getOrDefault(k, true)); }); } subscription = StoreEvents.subscribe(_ -\u0026gt; getUI() .ifPresent(ui -\u0026gt; ui.access(this::safeRefresh))); } Within the try block, suppressRefresh is set to true; all grid operations run without immediate data retrieval. Only when leaving the block and all column visibilities are set consistently does the guard provide a single final refresh with refreshAfter = true. As a result, the view is loaded exactly once after the first setup, rather than flickering through intermediate states during initialisation.\nsafeRefresh() also demonstrates its strengths when navigating between pages. The buttons for scrolling forward and backwards only change the internal page cursor and explicitly delegate the data update to the central method:\nprevBtn.addClickListener(_ -\u0026gt; { if (currentPage \u0026gt; 1) { currentPage--; refreshPageInfo(); safeRefresh(); } }); nextBtn.addClickListener(_ -\u0026gt; { int size = Optional.ofNullable(pageSize.getValue()).orElse(25); int maxPage = Math.max(1, (int) Math.ceil((double) totalCount / size)); if (currentPage \u0026lt; maxPage) { currentPage++; refreshPageInfo(); safeRefresh(); } }); pageSize.addValueChangeListener(e -\u0026gt; { currentPage = 1; grid.setPageSize(Optional.ofNullable(e.getValue()).orElse(25)); safeRefresh(); }); Here, the use of the guard is deliberately avoided, as each of these actions constitutes a clearly demarcated, independent refresh. Scrolling back or forward to a new page section, or flipping forward, should trigger a change in page size immediately. Nevertheless, all accesses continue to run via safeRefresh(), so that the mechanism remains centrally controllable. If the architecture changes in the future, an adjustment at this point is sufficient to modify the behaviour of the entire view consistently.\nTaken together, the combination of safeRefresh() and RefreshGuard transforms the refresh behaviour of the OverviewView from a hard-to-predict byproduct of many UI events to a controlled, deterministic strategy. Complex operations such as initialisation and reset are packed into closed, atomic blocks, while simple actions such as page changes and field changes are explicitly allowed to trigger a refresh. This gives the view both stability and transparency: readers of the code can clearly see where data updates occur, and users of the interface perceive a quiet, responsive application that responds to inputs in a comprehensible way.\nReset Mechanism: Full State-Clear # The reset mechanism of the OverviewView plays a special role within the search and filter architecture. It forms the fastest way back to a clearly defined, neutral initial state – a state in which neither search fragments nor extended filters, sorting options, nor deviating page sizes are active. While earlier implementations often reset only partial aspects, the revised version takes a consistently holistic approach: clicking \u0026ldquo;Reset\u0026rdquo; deletes all user-changed parameters without exception and resets the view as if it were being opened for the first time.\nConceptually, this mechanism is directly integrated into the previously introduced refresh architecture. Since the reset involves a large number of individual steps – emptying text fields, resetting checkboxes and date entries, restoring the default sorting, resetting the page size and closing the Advanced area – each of these actions would immediately trigger a refresh when viewed in isolation. To prevent this, and to treat all changes as logical operations, the entire reset process is embedded within a RefreshGuard.\nThe reset button implementation directly reflects these considerations. The listener for the reset button encapsulates all the necessary steps in a single, clearly defined block:\nresetBtn.addClickListener(_ -\u0026gt; { try (var _ = new RefreshGuard(true)) { globalSearch.clear(); codePart.clear(); codeCase.clear(); urlPart.clear(); urlCase.clear(); fromDate.clear(); fromTime.clear(); toDate.clear(); toTime.clear(); sortBy.clear(); dir.clear(); pageSize.setValue(25); sortBy.setValue(\u0026#34;createdAt\u0026#34;); dir.setValue(\u0026#34;desc\u0026#34;); currentPage = 1; searchScope.setValue(\u0026#34;URL\u0026#34;); advanced.setOpened(false); setSimpleSearchEnabled(true); globalSearch.focus(); } }); The reset process begins by clamping a RefreshGuard with the parameter set to true. This first sets the suppressRefresh flag, so that all safeRefresh() calls indirectly triggered in this block ‑remain ineffective. Only when leaving the block (controlled by refreshAfter = true) is a final refresh executed, making the cumulative new state visible in the grid.\nWithin the block, all user inputs are systematically returned to their original state. First, all search and filter fields are cleared: the global search field, the shortcode and URL fragments in the Advanced area, and the case-sensitivity checkboxes. The date and time fields for the time window under consideration are then set to zero. This ensures that no old period inadvertently affects later requests.\nIn the next step, the sorting is reset to its default values. First, sortBy and dir are removed to avoid potential inconsistencies; then, createdAt is explicitly set as the sort field and desc as the sort direction. The page size is also deliberately set back to the default value of 25 entries per page, and the page cursor currentPage is set to one. This creates a state that corresponds to the first time you enter the view: no running filters, a defined sorting and a comprehensible page setting.\nThe global search logic is also reinitialised. The search scope is reset to URL, and the Advanced scope is closed by Advanced.setOpened(false ). Calling setSimpleSearchEnabled(true) re-enables simple mode, and globalSearch.focus() ensures that the cursor lands directly in the global search field. This results in an intuitive process for the user: After the reset, he sees a neutral overview and can immediately start a new, simple search.\nThis keeps the user interface completely quiet during the reset: no flickering, no multiple queries, no inconsistent intermediate layout. Only when the entire process is complete is a single, final refresh executed, which establishes a consistent initial state across the grid. This stability is not only crucial for the user experience but also facilitates extensibility, as additional reset steps can be added without introducing new side effects. As long as new fields are included in this guard block, they remain part of the same atomic operation.\nIn combination with the search and filter logic, this results in a reset mechanism that is both semantically and technically cleanly modelled. Semantic because the user has a clear expectation – \u0026ldquo;everything back to square one\u0026rdquo; – that is fully met. Technically, because the mechanism is embedded in the central refresh architecture by the RefreshGuard, it causes neither uncontrolled side effects nor hidden data retrievals. On this basis, subsequent chapters can now address error cases, validations, and logging in a more granular way without touching the basic reset path again.\nError handling, validation, and robustness # With the growing number of functions in OverviewView, robustness is increasingly critical. The user interface should not only work reliably in ideal scenarios but also handle incomplete, contradictory, or incorrect inputs. The combination of global search, extended filter area, time windows, sorting settings and page size in particular presents the system with the challenge of recognising and stably handling even complex and potentially conflict-prone states.\nIn the revised architecture, robustness is not treated as an afterthought, but as an integral part of the UI logic. Many validation and error-avoidance mechanisms are deeply embedded in interaction points: in ValueChange listeners, when switching between Simple and Advanced modes, and when resetting and deriving a binding search string. The view does not aim to take away all user freedom, but instead offers a controlled environment in which only consistent states can arise. The technical side deliberately avoids complex error messages in favour of clear, deterministic rules that become directly visible in the interaction of the input elements.\nA central element of this robust logic is the automatic verification of consistency for the search and filter fields. The pattern of defensive synchronisation is already clearly evident in the global search field:\nglobalSearch.addValueChangeListener(e -\u0026gt; { var v = Optional.ofNullable(e.getValue()).orElse(\u0026#34;\u0026#34;); if (searchScope.getValue().equals(\u0026#34;Shortcode\u0026#34;)) { codePart.setValue(v); urlPart.clear(); } else { urlPart.setValue(v); codePart.clear(); } }); This logic ensures that the shortcode and URL fragment cannot be active simultaneously. As soon as the user enters something in the global search box, the value is interpreted as either a shortcode or a URL. The other field is consistently emptied. In this way, there are no conflicting filter states that would force the application to fulfil two search intentions simultaneously.\nThe Scope Selection listener reinforces this rule by ensuring that even subsequent changes to the search scope always result in a consistent state:\nsearchScope.addValueChangeListener(_ -\u0026gt; { var v = Optional.ofNullable(globalSearch.getValue()).orElse(\u0026#34;\u0026#34;); if (\u0026#34;Shortcode\u0026#34;.equals(searchScope.getValue())) { codePart.setValue(v); urlPart.clear(); } else { urlPart.setValue(v); codePart.clear(); } }); This prevents a user from typing a search query in URL mode, then switching to shortcode, thereby implicitly creating an invalid search model. The UI detects this state early and maps it to a clear, comprehensible model.\nThe validation and cleanup mechanisms are particularly evident in advanced mode when deriving a valid simple search state. The applyAdvancedToSimpleAndReset() method is the technical condensation of this approach:\nprivate void applyAdvancedToSimpleAndReset() { String code = Optional.ofNullable(codePart.getValue()).orElse(\u0026#34;\u0026#34;).trim(); String url = Optional.ofNullable(urlPart.getValue()).orElse(\u0026#34;\u0026#34;).trim(); final boolean hasCode = !code.isBlank(); final boolean hasUrl = !url.isBlank(); final String winnerValue = hasCode ? code : (hasUrl ? url : \u0026#34;\u0026#34;); final String winnerScope = hasCode ? \u0026#34;Shortcode\u0026#34; : \u0026#34;URL\u0026#34;; try (var _ = new RefreshGuard(true)) { codePart.clear(); codeCase.clear(); urlPart.clear(); urlCase.clear(); fromDate.clear(); fromTime.clear(); toDate.clear(); toTime.clear(); sortBy.clear(); dir.clear(); sortBy.setValue(\u0026#34;createdAt\u0026#34;); dir.setValue(\u0026#34;desc\u0026#34;); searchScope.setValue(winnerScope); if (!winnerValue.isBlank()) { globalSearch.setValue(winnerValue); } else { globalSearch.clear(); } setSimpleSearchEnabled(true); globalSearch.focus(); } } Several basic principles of robustness are intertwined here. First, it checks whether a shortcode or a URL fragment is set. If both are present, the shortcode fragment is given priority – a clear and comprehensible rule that avoids ambiguity. The entire Advanced area is then consistently cleaned up to ensure that no old or partially set values are unintentionally included in future filters.\nIn addition, RefreshGuard plays a special role in robust processes. Without the guard, the numerous changes within this method would trigger a series of refresh events. However, the guard suppresses these events selectively and triggers exactly one consistent refresh at the end. This prevents flickering UI transitions and ensures the user always sees the final state.\nAnother important component is validating time windows. The combination of DatePicker and TimePicker can naturally generate incomplete entries – such as a set date without time or vice versa. The logic in the backend transport takes care of these cases, but already in the UI code, a defensive determination of the timestamp prevents potential errors:\nprivate Optional\u0026lt;Instant\u0026gt; combineDateTime(DatePicker date, TimePicker time) { var d = date.getValue(); var t = time.getValue(); if (d == null \u0026amp;\u0026amp; t == null) return Optional.empty(); if (d == null) return Optional.empty(); var lt = (t != null) ? t : LocalTime.MIDNIGHT; return Optional.of(lt.atDate(d).atZone(ZoneId.systemDefault()).toInstant()); } The method is generous, but clear: a time value without a date is not a valid filter. In case of doubt, times are set to midnight, which keeps the model stable even in incomplete scenarios. This type of defensive modelling prevents incomplete UI inputs from leading to inconsistent backend requests.\nThe event handlers provide additional technical protection for paging and navigation. Actions such as scrolling between pages or changing the page size have a direct effect on the database, but should not trigger any unexpected side effects. The consistent use of safeRefresh() ensures that these changes only take effect if the refresh context allows it:\npageSize.addValueChangeListener(e -\u0026gt; { currentPage = 1; grid.setPageSize(Optional.ofNullable(e.getValue()).orElse(25)); safeRefresh(); }); Here, too, robustness is created by clear rules: A new page size resets the page cursor and triggers a controlled reload – never several, never via a detour.\nFinally, logging also contributes significantly to diagnostic robustness. In many places in the code, logger().info() is deliberately used to signal when search changes, refresh processes, or state transitions occur. These traces enable precise reconstruction of complex error patterns in UI-backend interactions without requiring additional debugging mechanisms.\nThe result is a system that does not rely on errors that are subsequently intercepted, but on deliberately modelled, conflict-free states. The user guidance is designed to prevent invalid or contradictory situations, and the technical foundation ensures that even incomplete or ambiguous inputs are converted into stable, controlled processes. Thus, the combination of validation, synchronisation, and protection mechanisms provides a viable basis for further expansion of the application.\nConclusion and outlook # The revision to the OverviewView is more than a collection of individual improvements. It marks a structural change that fundamentally reshapes the interaction among search, filtering, data updates, and user interaction. From an initially heterogeneous interface with scattered responsibilities, a clearly modelled, consistent and technically robust view has emerged, whose behaviour is comprehensible and extensible in all essential dimensions.\nA key result of this revision is the standardisation of the search logic. The introduction of a global search box, together with scope selection, forms a small, self-contained state machine that provides an intuitive entry point for the user. Complemented by the extended filter area, a flexible system is created that supports both fast search queries and more complex filter combinations – without competing with each other. The clear switch between advanced and straightforward mode prevents contradictory states and keeps cognitive effort low.\nEqually important is the redesigned refresh architecture. With safeRefresh() and the RefreshGuard, a mechanism has been introduced to stabilise the application\u0026rsquo;s refresh behaviour. Complex operations such as initialisation or reset are bundled into atomic, deterministic processes, while simple interactions can still react directly. This pattern operates in the background and is particularly noticeable to the user when the operation is quieter and less erratic.\nThe grid itself has also been further developed in terms of functionality and ergonomics. Copy functions, context-sensitive opening of details, dynamic flow badges and an improved column layout transform the table into an interactive workspace that not only provides information, but also enables action. This proximity between data and interaction reduces the need for additional dialogue changes, thereby contributing to a more fluid workflow.\nThe robustness of the view ultimately results from a variety of small but effective mechanisms: automatic synchronisation of filter fields, defensive evaluation of incomplete entries, clear prioritisation rules, and consistent logging. All these aspects ensure that the application remains reliable even under unusual input combinations and that the causes of errors can be traced if necessary.\nThis structural basis enables broad future expansion. The clear separation of UI logic, filter model, and refresh strategy provides a stable foundation on which subsequent features can be built without breaking points. Among other things, the following are conceivable:\n– a server-side full-text search that extends the Simple/Advanced model, – colour or iconographic markings of other states such as \u0026ldquo;soon to expire\u0026rdquo;, – bulk actions for multiple selections, – a modular sorting and filtering pipeline, – tagging or labelling functions for short URLs, – advanced column settings or custom views.\nThe new OverviewView thus not only represents an improvement over the status quo but also marks a structural turning point. It creates clarity where implicit or scattered logic previously prevailed and establishes mechanisms that keep the system stable over the long term. In its entirety, the revision represents an important step towards a modern, scalable, and maintainable UI architecture that meets the requirements of growing use cases.\nCheers Sven\n","date":"14 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-from-ui-interactions-to-a-deterministic-refresh-architecture/","section":"Posts","summary":"After the first part explained the conceptual basics and the new interactions among global search, search scopes, and advanced filters, the second part focuses on the technical mechanisms that enable these interactions. It is only the revised refresh architecture – above all the interaction of safeRefresh() and RefreshGuard – that ensures that the OverviewView remains calm, deterministic and predictable despite numerous potential triggers.\n","title":"Advent Calendar 2025 - From UI Interactions to a Deterministic Refresh Architecture","type":"posts"},{"content":"Since its inception, the URL shortener\u0026rsquo;s continuous development has focused on two core goals: a robust technical foundation without external frameworks and a modern, productive user interface that is both intuitive and efficient for power users. As part of the current development stage, an essential UI module has been revised – the OverviewView, i.e. the view in which users search, filter and manage all saved shortenings.\nIn the previous version, it became increasingly clear that the search and filtering components were decoupled. The basic search offered limited functionality, while the advanced filters were present but not integrated into a real control flow. In addition, the interaction between user actions such as reset, filter changes, paging, or opening the detail view was not sufficiently stabilised, which sometimes led to multiple refreshes or contradictory UI states.\nThe revision now implemented aims to achieve several structural objectives. The central focus was on standardising interactions: the search function was redesigned and integrated into a unified concept, including a global search bar with a clearly defined scope and a structured area for advanced filters. In parallel, the technical architecture of the refresh mechanisms has been revised to deliver a quieter, more reliable user experience.\nThis introduction first outlines the rationale for the changes and situates them within the broader development context. In the following chapters, the new global search is first described, followed by the synchronisation mechanisms, extended filters, internal refresh architecture, and grid improvements. The goal is not only a functional description but also a technical analysis of the mechanisms that enable these improvements and underpin future expansions.\nThe source code for this development step can be found on GitHub\nand can be found here:https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-07\nThe new global search # The introduction of global search marks a central step towards a consistent yet flexible operating logic within the OverviewView. While in the previous version the search function consisted of several independent elements, a uniform interaction surface has now been created, whose behaviour is clearly defined and technically cleanly modelled. The basis is a single text field, supplemented by a search-area selection, which eliminates the prior separation between URL and shortcode searches.\nIn the source code, this standardisation is first reflected in the explicit introduction of two central UI components, which are declared as fixed components of the view:\nprivate final TextField globalSearch = new TextField(); private final ComboBox\u0026lt;String\u0026gt; searchScope = new ComboBox\u0026lt;\u0026gt;(\u0026#34;Search in\u0026#34;); private final Button advancedBtn = new Button(\u0026#34;Advanced filters\u0026#34;, new Icon(VaadinIcon.SLIDERS)); This clearly defines that there is precisely one global search field and exactly one selection for the search area. Both components are created early in the view\u0026rsquo;s life cycle and are therefore available to all subsequent configuration steps. The actual design is done in the buildSearchBar() method, in which placeholders, width, and interaction behavior are specifically defined:\nprivate Component buildSearchBar() { globalSearch.setPlaceholder(\u0026#34;Search all...\u0026#34;); globalSearch.setClearButtonVisible(true); globalSearch.setWidth(\u0026#34;28rem\u0026#34;); globalSearch.setValueChangeMode(LAZY); globalSearch.setValueChangeTimeout(VALUE_CHANGE_TIMEOUT); searchScope.setItems(\u0026#34;URL\u0026#34;, \u0026#34;Shortcode\u0026#34;); searchScope.setValue(\u0026#34;URL\u0026#34;); searchScope.setWidth(\u0026#34;11rem\u0026#34;); pageSize.setMin(1); pageSize.setMax(500); pageSize.setStepButtonsVisible(true); pageSize.setWidth(\u0026#34;140px\u0026#34;); // ... } The global search is not treated as an arbitrary text field, but is deliberately modelled as a central control element. The placeholder \u0026ldquo;Search all\u0026hellip;\u0026rdquo; makes it clear that, regardless of the specific search area, the user initially enters only one search term. The actual routing of this value into the appropriate technical field handles the logic associated with the search field and scope selection. Choosing LAZY mode, combined with an explicit timeout, ensures that server-side filter requests are triggered only when the user has completed input.\nThe search box serves as the starting point for all queries. As soon as the user enters, the content is assigned to either the URL or the shortcode component of the filter model, depending on the currently selected search area. This assignment is not only a UI-side mechanism but also a clear rule within the internal logic: the value of the global search field is always bound to exactly one of the two fields in the request object. The central implementation of this coupling is carried out via the ValueChangeListener of the global search field:\nglobalSearch.addValueChangeListener(e -\u0026gt; { var v = Optional.ofNullable(e.getValue()).orElse(\u0026#34;\u0026#34;); if (searchScope.getValue().equals(\u0026#34;Shortcode\u0026#34;)) { codePart.setValue(v); urlPart.clear(); } else { urlPart.setValue(v); codePart.clear(); } }); Here, each new value is first converted to a safe, non-null variant. The value of the ComboBox searchScope then decides whether the search term is interpreted as a shortcode filter (codePart) or a URL filter (urlPart). Only one of the two fields may be occupied at a time; the other is cleared automatically. This avoids ambiguous search situations in which multiple filters are applied simultaneously and uncoordinatedly. The global search field thus becomes the sole source for exactly one specific logical filter state.\nAnother major innovation is the tight coupling between the search field and the search area. The selection of the range – URL or shortcode – directly determines which part of the filter model is active. To ensure that this relationship remains consistent in both directions, the scope selection also reacts to changes and reflects the current search value in the appropriate field:\nsearchScope.addValueChangeListener(_ -\u0026gt; { var v = Optional.ofNullable(globalSearch.getValue()).orElse(\u0026#34;\u0026#34;); if (\u0026#34;Shortcode\u0026#34;.equals(searchScope.getValue())) { codePart.setValue(v); urlPart.clear(); } else { urlPart.setValue(v); codePart.clear(); } }); While the text field listener reacts when the search term changes, it is responsible for maintaining consistent search state when the user subsequently changes the search scope. Both implementations follow the same pattern: a standard source value is interpreted as either a shortcode or a URL, and the inactive field is consistently emptied. This keeps the search interface not only visually clear, but also logical.\nThe behaviour of the global search is also designed to enable clear prioritisation when combined with the advanced filters. As long as the advanced filters are closed, the global search box controls the filter state independently. When the user opens the advanced area, the global search loses its active role and is relegated to the background, both visually and technically. This separation is encapsulated by a small helper method that explicitly determines the state in which the simple search may be:\nprivate void setSimpleSearchEnabled(boolean enabled) { globalSearch.setEnabled(enabled); searchScope.setEnabled(enabled); resetBtn.setEnabled(true); globalSearch.setHelperText(enabled ? null : \u0026#34;Disabled while Advanced filters are open\u0026#34;); } This method not only controls the activation and deactivation of the input elements but also provides context-sensitive help text that explains why global search is unavailable in open Advanced mode. This prevents two parallel filter sources from competing with each other and destabilizing the overall state. At the same time, the operating concept remains transparent, as the UI clearly communicates its status.\nWith this revamped global search, an intuitive, clear entry point has been created with a well-defined function from both the user and technical architecture perspectives. The search field, the scope selection, and the associated filter fields form a small, self-contained state machine whose behaviour is explicitly modelled in the code. The following chapters now examine how this search interacts with the other components and how the underlying synchronisation logic ensures consistent state transitions.\nSearch scopes and synchronisation logic # While the global search offers a clear entry point, its real strength only becomes apparent through interaction with the underlying synchronisation logic. It is crucial that the selected search scope – i.e. the decision between URL and shortcode – does not remain just a visual interface detail, but is consistently transferred to the internal state model. The goal of this layer is to ensure that at any given time, it is clear which filter is active and which parts of the UI represent that filter.\nFrom the user\u0026rsquo;s perspective, the behaviour can be divided into two central scenarios. In simple mode, the combination of the global search field and scope selection directly controls the filter state. The user implicitly determines, via the input context, whether to search for a target URL or a shortcode. In advanced mode, by contrast, the global search is reversed, and the detail fields fully control the filter state. This transition between modes is the core of synchronisation logic.\nFrom a technical standpoint, this logic is based on a few, clearly defined principles. First, there is exactly one source for the effective search string at any given time. In simple mode, this is the global search field, which is mapped to either the URL or the shortcode component of the filter model, depending on the scope. In advanced mode, the dedicated URL and shortcode fields in the Advanced area are transferred directly to the request object. Second, there must be no competing states: if the user is working in Advanced mode, global search items are disabled; when the user returns to simple mode, the state is derived from the previous detail values.\nThe technical implementation of this switch begins where the View establishes the Advanced area as a controlling element. Central is the listener, which reacts to the opening and closing of the details container:\nadvanced.addOpenedChangeListener(ev -\u0026gt; { boolean nowClosed = !ev.isOpened(); if (nowClosed) { applyAdvancedToSimpleAndReset(); } else { setSimpleSearchEnabled(false); } }); These few lines model the entire state change between the modes. When the user opens the Advanced section, the simple search is disabled. When closing, not only is the Advanced area collapsed, but a consolidation step is also triggered via applyAdvancedToSimpleAndReset(), which converts the previous detailed configuration into a simple, global search state.\nTo ensure that disabling the simple search does not lead to an inconsistent UI impression, the View encapsulates the necessary adjustments in a small helper method:\nprivate void setSimpleSearchEnabled(boolean enabled) { globalSearch.setEnabled(enabled); searchScope.setEnabled(enabled); resetBtn.setEnabled(true); globalSearch.setHelperText(enabled ? null : \u0026#34;Disabled while Advanced filters are open\u0026#34;); } The method controls both the interactivity of the search field and scope selection as well as the accompanying help text. Once the Advanced section is opened, the Basic Search values are retained, but cannot be changed. At the same time, the helper text makes it clear that global search is currently disabled. At this level, the first part of the principle described above is implemented: There is always only one active source that determines the effective filter state.\nThe opposite direction – from advanced mode back to simple mode – is more complex because a choice has to be made here. In the Advanced area, a shortcode fragment and a URL fragment can be entered at the same time. Both would be suitable as filters but cannot be readily combined into a single global search field. This is precisely where applyAdvancedToSimpleAndReset() comes in:\nprivate void applyAdvancedToSimpleAndReset() { String code = Optional.ofNullable(codePart.getValue()).orElse(\u0026#34;\u0026#34;).trim(); String url = Optional.ofNullable(urlPart.getValue()).orElse(\u0026#34;\u0026#34;).trim(); final boolean hasCode = !code.isBlank(); final boolean hasUrl = !url.isBlank(); final String winnerValue = hasCode ? code : (hasUrl ? url : \u0026#34;\u0026#34;); final String winnerScope = hasCode ? \u0026#34;Shortcode\u0026#34; : \u0026#34;URL\u0026#34;; try (var _ = new RefreshGuard(true)) { codePart.clear(); codeCase.clear(); urlPart.clear(); urlCase.clear(); fromDate.clear(); fromTime.clear(); toDate.clear(); toTime.clear(); sortBy.clear(); dir.clear(); sortBy.setValue(\u0026#34;createdAt\u0026#34;); dir.setValue(\u0026#34;desc\u0026#34;); searchScope.setValue(winnerScope); if (!winnerValue.isBlank()) { globalSearch.setValue(winnerValue); } else { globalSearch.clear(); } setSimpleSearchEnabled(true); globalSearch.focus(); } } The method begins by evaluating the codePart and urlPart detail fields. Both values are defensively converted to strings and then checked for non-empty content. Two things are derived from this: a \u0026ldquo;winner\u0026rdquo; value and a \u0026ldquo;winner\u0026rdquo; scope. If a shortcode fragment is set, it takes precedence over any URL fragment. Only if there is no shortcode and only a URL is the URL considered a winner. If both are empty, an empty search string is used, and the scope is reset to \u0026ldquo;URL\u0026rdquo;. In this way, the prioritisation described in the running text is implemented in practice, without ambiguity.\nIn the second block of the method, all Advanced fields are consistently reset. In addition to the text fields for shortcodes and URLs, this applies to the case-sensitivity checkboxes and the time-slot and sorting fields. The Advanced range is thus returned to a defined initial state. Thanks to the RefreshGuard, this reset does not occur through multiple individual refreshes; instead, it is treated as an aggregated state change that culminates in a controlled reload.\nOnly then is the previously determined winner state reflected into the simple search. The global search scope is set to winnerScope; the global search string is either filled with winnerValue or left empty. Finally, the simple search is reactivated, and the focus is set to the global search field. This provides the user with a straightforward, reduced interface after closing the Advanced area, reflecting the active filter state derived from the previously selected detail values.\nOverall, this yields a small but precise state machine. Opening the Advanced pane shifts control entirely to the detail fields and visibly disables global search. Closing triggers a controlled reduction to a single, easy-to-understand filter state. The synchronisation logic remains fully comprehensible in the code, is encapsulated in a few clearly structured methods, and can be extended with additional filter fields if necessary without violating the basic principle. On this basis, the following chapters can now examine other aspects of filtering in detail, such as the extended filter fields and the refresh architecture.\nAdvanced Filters: Conception and UI Design # The global search serves as a compact entry point to the OverviewView\u0026rsquo;s filter logic. However, their range of functions is deliberately limited to ensure a low barrier to entry and rapid operation. As soon as the requirements go beyond a simple text fragment, this model reaches its natural limits. This is where Advanced Filters come into play, giving users much finer control over short URL filtering while providing a structured, visually comprehensible interface.\nConceptually, the Advanced area was designed as a deliberately separate mode. It should not be understood as a mere extension of the existing search field, but rather as an independent filter context that becomes active only when the user explicitly opens it. This decision takes into account two considerations. On the one hand, the simple mode should not be burdened with options unnecessary in many everyday scenarios. On the other hand, advanced filter operations – such as the combination of a shortcode fragment, a URL substring, a time window, and sorting – should have a clearly identifiable workspace in which all associated input elements are spatially bundled.\nIn the source code, this concept already begins at the field level, where the components for the Advanced area are clearly declared separate from the global search:\nprivate final TextField codePart = new TextField(\u0026#34;Shortcode contains\u0026#34;); private final Checkbox codeCase = new Checkbox(\u0026#34;Case-sensitive\u0026#34;); private final TextField urlPart = new TextField(\u0026#34;Original URL contains\u0026#34;); private final Checkbox urlCase = new Checkbox(\u0026#34;Case-sensitive\u0026#34;); private final DatePicker fromDate = new DatePicker(\u0026#34;From (local)\u0026#34;); private final TimePicker fromTime = new TimePicker(\u0026#34;Time\u0026#34;); private final DatePicker toDate = new DatePicker(\u0026#34;To (local)\u0026#34;); private final TimePicker toTime = new TimePicker(\u0026#34;Time\u0026#34;); private final ComboBox\u0026lt;String\u0026gt; sortBy = new ComboBox\u0026lt;\u0026gt;(\u0026#34;Sort by\u0026#34;); private final ComboBox\u0026lt;String\u0026gt;dir = new ComboBox\u0026lt;\u0026gt;(\u0026#34;Direction\u0026#34;); These fields define the semantic dimensions of the advanced filters: textual filtering via shortcode and original URL (optional), an explicit time window, and sorting criteria. The fact that they are listed as separate attributes of the view expresses the separate mode described above: they do not belong to the global search but to an individual, extended view of the data space.\nFrom a UI perspective, this separation manifests as a details container that makes the advanced filters collapsible. When closed, the Advanced section does not occupy any additional space and only indicates, via its header, that more options are available. Only when opened does a structured form unfold, which arranges the various filter dimensions into logically related groups. The concrete design begins with the configuration of the fields themselves:\ncodePart.setPlaceholder(\u0026#34;e.g. ex-\u0026#34;); codePart.setValueChangeMode(LAZY); codePart.setValueChangeTimeout(VALUE_CHANGE_TIMEOUT); codePart.addValueChangeListener(_ -\u0026gt; safeRefresh()); urlPart.setPlaceholder(\u0026#34;e.g. docs\u0026#34;); urlPart.setValueChangeMode(LAZY); urlPart.setValueChangeTimeout(VALUE_CHANGE_TIMEOUT); urlPart.addValueChangeListener(_ -\u0026gt; safeRefresh()); sortBy.setItems(\u0026#34;createdAt\u0026#34;, \u0026#34;shortCode\u0026#34;, \u0026#34;originalUrl\u0026#34;, \u0026#34;expiresAt\u0026#34;); dir.setItems(\u0026#34;asc\u0026#34;, \u0026#34;desc\u0026#34;); fromDate.setClearButtonVisible(true); toDate.setClearButtonVisible(true); fromTime.setStep(Duration.ofMinutes(15)); toTime.setStep(Duration.ofMinutes(15)); fromTime.setPlaceholder(\u0026#34;hh:mm\u0026#34;); toTime.setPlaceholder(\u0026#34;hh:mm\u0026#34;); The text fields for the shortcode and URL provide meaningful placeholders and, like the global search, use a lazy ValueChange mode with a timeout. This prevents a new filter run from being triggered immediately with each input, while at the same time the filters respond quickly to changes. The sorting pair sortBy is preassigned to you, with permissible values, and thus is embedded within a defined space of possible sorting strategies. Convenience functions supplement the date and time fields: Clear buttons, fixed 15-minute time grids, and placeholders for the time format help users enter data while reducing the risk of invalid values.\nThe spatial structure of Advanced Filters is designed to visually group related information. Instead of placing all components on a single long line, the implementation supports multiple upstream layouts. First, the date and time fields are merged into two groups:\nvar fromGroup = new HorizontalLayout(fromDate, fromTime); fromGroup.setDefaultVerticalComponentAlignment(Alignment.END); var toGroup = new HorizontalLayout(toDate, toTime); toGroup.setDefaultVerticalComponentAlignment(Alignment.END); The two horizontal layouts, „fromGroup“ and „toGroup“, ensure that date and time visually appear as a coherent unit. The vertical alignment at the bottom creates a calm, uniform appearance, even when field heights vary slightly. These groups are then embedded in a FormLayout, which forms the actual responsive structure:\nFormLayout searchBlock = new FormLayout(); searchBlock.setWidthFull(); searchBlock.add(codePart, urlPart, new HorizontalLayout(codeCase, urlCase)); searchBlock.add(fromGroup, toGroup); searchBlock.setResponsiveSteps( new FormLayout.ResponsiveStep(\u0026#34;0\u0026#34;, 1), new FormLayout.ResponsiveStep(\u0026#34;32rem\u0026#34;, 2), new FormLayout.ResponsiveStep(\u0026#34;56rem\u0026#34;, 3) ); Here, the textual filters – shortcode and URL – as well as the associated case sensitivity checkboxes are arranged together in a block. Below this are the groups for the time slot. ResponsiveSteps determines how many columns the layout may use at different widths. Below 32 rem, a single-column layout is selected; at medium width, two columns are available; and above 56 rem, the layout can be extended to three columns. In this way, the input mask remains readable and well-structured across wide desktop views and narrower windows or split screens.\nThe sorting control is deliberately visually decoupled from the search block, but positioned on the same horizontal axis. For this purpose, a separate toolbar area will be set up:\nsortBy.setLabel(null); sortBy.setPlaceholder(\u0026#34;Sort by\u0026#34;); sortBy.setWidth(\u0026#34;12rem\u0026#34;); dir.setLabel(null); dir.setPlaceholder(\u0026#34;Direction\u0026#34;); dir.setWidth(\u0026#34;8rem\u0026#34;); HorizontalLayout sortToolbar = new HorizontalLayout(sortBy, dir); sortToolbar.setAlignItems(FlexComponent.Alignment.END); By removing the ComboBox labels and using placeholders, the interface remains compact without sacrificing intelligibility. The fixed width ensures stable alignment, while the horizontal grouping makes it clear that both fields functionally belong together. The alignment at the bottom blends with the rest of the header, where the filter boxes are also aligned on a common baseline.\nFinally, the search block and sort bar are merged into a single header layout that constitutes the visible content of the Advanced area. This header layout is then embedded in a Details component:\nHorizontalLayout advHeader = new HorizontalLayout(searchBlock, sortToolbar); advHeader.setWidthFull(); advHeader.setSpacing(true); advHeader.setAlignItems(FlexComponent.Alignment.START); advHeader.expand(searchBlock); advHeader.getStyle().set(\u0026#34;flex-wrap\u0026#34;, \u0026#34;wrap\u0026#34;); advHeader.setVerticalComponentAlignment(FlexComponent.Alignment.END, sortToolbar); advanced = new Details(\u0026#34;Advanced filters\u0026#34;, advHeader); advanced.setOpened(false); advanced.getElement().getThemeList().add(\u0026#34;filled\u0026#34;); setSimpleSearchEnabled(!advanced.isOpened()); The advHeader ensures the search block occupies the available space, while the sorting tools are anchored to the right edge. Enabling flex-wrap allows the header to wrap to multiple lines within a limited width without disrupting the logical proximity of the elements. The Details component includes this header and makes the entire Advanced section collapsible. When closed, only the title \u0026ldquo;Advanced filters\u0026rdquo; remains visible; when opened, the complete form unfolds. The application of the filled theme also provides the area with a visual demarcation from the surrounding layout.\nIn terms of content, the Advanced Filters design prioritises making the essential dimensions of the data directly accessible. Shortcodes and destination URLs serve as textual entry points, and the search can be configured as case-sensitive or case-insensitive. The temporal context of the short URL – such as the creation or expiration interval considered – is mapped via combined date and time fields that explicitly work in the user\u0026rsquo;s local context. Finally, the sort field and sorting direction provide fine control over the order of displayed entries, enabling newly created or soon-expiring links to be brought to the foreground.\nThis expanded range of functions should not leave the user confronted with a confusing number of control elements. The precise spatial separation within the hinged container, the well-thought-out grouping of the fields, and the responsive arrangement therefore not only provide an aesthetic but, above all, a cognitive relief. The user can choose whether to use the standard global search options or switch to expert mode, which provides more detailed control over the database.\nOn this basis, the Advanced area can be seamlessly embedded into the rest of the architecture in the following chapters. In particular, the connection between the synchronisation logic and the refresh architecture described above demonstrates how the UI design and the internal state engine work in concert to keep even more complex filter requests stable and easy to understand.\nCheer Sven\n","date":"13 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-from-simple-search-to-expert-mode-advanced-filters-and-synchronised-scopes-for-power-users/","section":"Posts","summary":"Since its inception, the URL shortener’s continuous development has focused on two core goals: a robust technical foundation without external frameworks and a modern, productive user interface that is both intuitive and efficient for power users. As part of the current development stage, an essential UI module has been revised – the OverviewView, i.e. the view in which users search, filter and manage all saved shortenings.\n","title":"Advent Calendar 2025 - From Simple Search to Expert Mode: Advanced Filters and Synchronised Scopes for Power Users","type":"posts"},{"content":"Today\u0026rsquo;s update introduces another practical development step for the URL shortener. After the last few days were dedicated to UI refinement and the better structuring of detailed dialogues and form logic, the focus is now on an aspect that plays a significant role in the everyday life of many users: the flexible management of multiple aliases per target URL.\nYou can find the source code for this development status on GitHub underhttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-06\nHere are the relevant screenshots of the Vaadin application.\nWhat was previously only possible sequentially – an alias, a destination address – will now become a dynamic workspace. Users can add new aliases to existing short links, create variants for different use cases, or manage parallel campaigns without creating new records each time. The MultiAliasEditorStrict forms the core of this new workflow: inline validation, fast feedback, clear status indicators and an event flow that propagates changes directly into the OverviewView. The application responds immediately to user input and keeps the entire interface consistent, without requiring manual reloads or context switching.\nBut this is much more than just a UI comfort update. The increased flexibility in the alias structure inevitably leads to higher demands on server-side stability – especially on the stability of forwarders. Once several aliases point to the same destination address, the redirect logic must be deterministic and reliably detect erroneous requests. Therefore, a consistent, centralised request check was introduced in parallel with the UI extension to secure all forwarding operations and ensure predictable HTTP behaviour.\nThese changes combine two sides of the same coin: greater user freedom in their daily work and a robust technical foundation that reliably supports it. In the following, we will therefore take a closer look at the revised redirect implementation and the new security mechanisms that underpin stable, traceable, and maintainable redirect behaviour.\nMore Robust Redirects and HTTP Security # In parallel with the extensive improvements to the user interface, the server-side component of the URL shortener has also been revised. A central focus was on increasing the stability and security of HTTP communication. Especially when handling redirects, several methods have had to carry out similar checks – for example, whether the correct HTTP method was used or whether a request was incomplete. These repetitions led to inconsistent behaviour and made maintenance difficult.\nIn the process, a consistent request-handling system was introduced to centralise these checks. The code for the RedirectHandler clearly shows how a robust redirect is now implemented:\npackage com.svenruppert.urlshortener.server.handler; import com.svenruppert.urlshortener.core.http.HttpStatus; import com.svenruppert.urlshortener.core.store.UrlMappingStore; import com.svenruppert.urlshortener.server.util.RequestMethodUtils; import com.svenruppert.urlshortener.server.util.ResponseWriter; import com.sun.net.httpserver.HttpExchange; import java.io.IOException; public class RedirectHandler implements HttpHandler, HasLogger { private final UrlMappingLookup store; public RedirectHandler(UrlMappingLookup store) { this.store = store; } @Override public void handle(HttpExchange exchange) throws IOException { if (! RequestMethodUtils.requireGet(exchange)) return; final String path = exchange.getRequestURI().getPath(); e.g. \u0026#34;/ABC123\u0026#34; if (path == null || !path.startsWith(PATH_REDIRECT)) { exchange.sendResponseHeaders(400, -1); return; } final String code = path.substring((PATH_REDIRECT).length()); if (code.isBlank()) { exchange.sendResponseHeaders(400, -1); return; } logger().info(\u0026#34;looking for short code {}\u0026#34;, code); Optional\u0026lt;String\u0026gt; target = store .findByShortCode(code) .map(ShortUrlMapping::originalUrl); if (target.isPresent()) { exchange.getResponseHeaders().add(\u0026#34;Location\u0026#34;, target.get()); exchange.sendResponseHeaders(302, -1); } else { exchange.sendResponseHeaders(404, -1); } } } The RedirectHandler class ensures that only valid GET requests are accepted. Invalid methods – such as POST or DELETE – are caught by the RequestMethodUtils helper class. This prevents inadvertent writes to a read-only resource.\nThe check logic in RequestMethodUtils is deliberately kept simple and centralises all recurring security checks. The following excerpt shows how she works internally:\npackage com.svenruppert.urlshortener.server.util; import com.svenruppert.urlshortener.core.http.HttpStatus; import com.sun.net.httpserver.HttpExchange; import java.io.IOException; public class RequestMethodUtils { public static boolean requireGet(HttpExchange exchange) throws IOException { HasLogger.staticLogger().info(\u0026#34;handle ... {} \u0026#34;, exchange.getRequestMethod()); Objects.requireNonNull(exchange, \u0026#34;exchange\u0026#34;); if (!\u0026#34; GET\u0026#34;.equalsIgnoreCase(exchange.getRequestMethod())) { writeJson(exchange, METHOD_NOT_ALLOWED, \u0026#34;Only GET is allowed for this endpoint.\u0026#34;); return false; } return true; } SNIP... } This helper class serves as a compact security wall: every request-processing method calls it at the beginning. This creates predictable, consistent behaviour across the HTTP stack. This is particularly relevant for forwarding, as unauthorised or erroneous methods are reliably blocked.\nResponse generation is handled by ResponseWriter, which handles clean header configuration and uniform error output. In contrast to earlier implementations, which manually generated JSON or text responses in several places, there is now a central method:\npublic static void writeJson(HttpExchange ex, HttpStatus httpStatus, String message) throws IOException { HasLogger.staticLogger().info(\u0026#34;writeJson {}, {}\u0026#34;, httpStatus, message); byte[] data = message.getBytes(UTF_8); Headers h = ex.getResponseHeaders(); h.set(CONTENT_TYPE, JSON_CONTENT_TYPE); ex.sendResponseHeaders(httpStatus.code(), data.length); try (OutputStream os = ex.getResponseBody()) { os.write(data); } catch (Exception e) { HasLogger.staticLogger().info(\u0026#34;writeJson {} (catch)\u0026#34;, e.getMessage()); byte[] body = (\u0026#34;{\\\u0026#34;error\\\u0026#34;:\\\u0026#34;\u0026#34; + e.getMessage() + \u0026#34;\\\u0026#34;}\u0026#34;).getBytes(StandardCharsets.UTF_8); try { ex.getResponseHeaders().set(CONTENT_TYPE, JSON_CONTENT_TYPE); ex.sendResponseHeaders(INTERNAL_SERVER_ERROR.code(), body.length); ex.getResponseBody().write(body); } catch (Exception ignoredI) { HasLogger.staticLogger().info(\u0026#34;writeJson (catch - ignored I) {} \u0026#34;, ignoredI.getMessage()); } } finally { try { ex.close(); } catch (Exception ignoredII) { HasLogger.staticLogger().info(\u0026#34;writeJson (catch - ignored II) {} \u0026#34;, ignoredII.getMessage()); } } } The merging of these helper classes yields a well-structured server architecture. Each handler is leaner, easier to read, and easier to test. For users, this results in a more reliable redirect: error messages are displayed consistently, HTTP status codes match the expected behaviour exactly, and unauthorised methods do not cause side effects.\nThis also gives the URL shortener additional robustness at the protocol level. Error handling is traceable, behaviour is transparent, and server components follow the same principles of clarity and reusability introduced in the UI. In this way, the system\u0026rsquo;s technical integrity is strengthened, as is user confidence in its stability.\nConclusion: User Comfort Meets Clean Design # With the completion of this development step, the URL shortener has developed into a much more mature system – both technically and conceptually. What started as a simple platform for creating and managing individual shortlinks has become a full-fledged application that balances user needs, security, and maintainability. The transition from a one-alias logic to a flexible multi-alias approach is probably the most visible progress. Still, the real strength of this update lies in the design\u0026rsquo;s consistent coherence.\nFrom the user\u0026rsquo;s perspective, a tool is now available that makes workflows more fluid, consistent, and traceable. The detail-dialogue and the Create View form a harmonious pair: both use the same editor, the exact validation mechanisms and the same event flow. Actions are immediately visible, and changes are automatically displayed. The result is a surface that hides its technical complexity and instead focuses on efficiency and clarity.\nA clear development is also noticeable on the architectural level. The interplay of Vaadin EventBus, component-based UI structure and centralised HTTP handling has resulted in a maintenance-friendly, extensible system. Every level – from the user interface to the validation logic to the server – follows the same principles: clear responsibilities, loose coupling, and transparent behaviour. This uniformity is evident not only in the code but above all in everyday use.\nThis step thus sets a clear benchmark for future project expansions. The concept of multiple aliases, the consistent reactivity of the UI, and the clean server design will not remain isolated features in the coming development phases; they will serve as a structural basis. Working on the user experience has shown that technical design and interaction quality are not mutually exclusive – on the contrary, they reinforce each other when they are consistently coordinated.\nCheers Sven\n","date":"12 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-introduction-of-multiple-aliases-part-2/","section":"Posts","summary":"Today’s update introduces another practical development step for the URL shortener. After the last few days were dedicated to UI refinement and the better structuring of detailed dialogues and form logic, the focus is now on an aspect that plays a significant role in the everyday life of many users: the flexible management of multiple aliases per target URL.\n","title":"Advent Calendar 2025 - Introduction of multiple aliases - Part 2","type":"posts"},{"content":" Introduction: More convenience for users # With today\u0026rsquo;s development milestone for the URL Shortener project, a crucial improvement has been achieved, significantly enhancing the user experience. Up to this point, working with short URLs was functional, but in many ways it was still linear: each destination URL was assigned exactly one alias. This meant users had to create a new short URL for each context or campaign, even when the destination was identical. While this approach was simple, it wasn\u0026rsquo;t sufficiently flexible to meet real-world application requirements.\nYou can find the source code for this development status on Github underhttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-06\nIn everyday life, users quickly reach their limits when they want to map the same destination address for different purposes – for example, to analyse access from other sources or to separate internal and external uses. Working with only one alias per target URL forced them to work around and led to unnecessary redundancy in the stored mappings. Especially in professional environments where tracking, traceability, and reusability are essential, this situation was unsatisfactory.\nWith today\u0026rsquo;s development, this restriction has been lifted. The system now allows multiple aliases to point to a common destination URL. This focuses on user convenience: Existing short URLs can now be extended with new aliases without creating a new dataset. The process is accompanied by an intuitive user interface that integrates seamlessly into the application\u0026rsquo;s detailed dialogue.\nThis extension is more than just a functional improvement. It changes the way users interact with the system. Whereas previously each alias represented a separate entity, a central destination URL with variable identifiers is now introduced. The user no longer thinks in terms of individual links, but rather in terms of a flexible alias space mapped to a destination address. This reduces cognitive load and noticeably accelerates typical workflows.\nOn a technical level, this development was supported by the introduction of new UI components and event mechanisms that enable dynamic synchronisation between the detail dialogue and the overview. But the real meaning lies in the user experience itself: the shortener becomes a tool that adapts to real-world usage patterns – not the other way around.\nFrom single alias to multi-alias management # The introduction of multi-alias management marks a clear turning point in the URL shortener\u0026rsquo;s development. While previously each short URL was inseparably bound to a single alias, the system now supports a more flexible, yet more natural, assignment model. From a technical standpoint, this means a structural expansion of the data model, but from the user\u0026rsquo;s perspective, it means one thing above all: freedom to manage one\u0026rsquo;s own short-link landscape.\nAt the heart of this innovation is the MultiAliasEditorStrict component, designed to intuitively guide users through managing multiple aliases. The editor lets you add new aliases, modify existing ones, or remove erroneous entries without opening modal dialogues or new dialogues. Instead, the user remains in a consistent context – the detailed view of a target URL – and can act directly there. This decision was deliberate to avoid friction in the interaction and maintain the flow between viewing and editing.\nThe implementation of MultiAliasEditorStrict follows a clear principle: strict validation with maximum user-friendliness. Each alias entered is immediately checked for empty values, duplicate entries, and syntactic correctness. Errors are visualised directly in the input field, providing the user with immediate feedback. This direct feedback builds trust in the input and prevents incomplete or erroneous data from reaching the persistence layer.\nA key feature of this component is its interaction with the application\u0026rsquo;s event infrastructure. As soon as a new alias is saved or an existing alias is changed, the editor triggers an event that propagates across the entire UI. This means that other views – such as the overview list – are also automatically updated without requiring a reload. This event control ensures a reactive user interface that always reflects the current state.\nFrom a technical perspective, multi-alias management illustrates how consistently implemented user orientation and clean architecture reinforce one another. The system remains modular, with components clearly separated, yet the user perceives a consistent, organic interface. What was previously a sequential process – create, check, correct – becomes an interactive dialogue with the system, which reacts to the user\u0026rsquo;s actions in real time.\nThus, MultiAliasEditorStrict is not only the heart of the new alias logic but also a harbinger of future developments. Its architecture lays the foundation for further interactions such as mass edits, alias groupings or time-controlled alias activations. In this sense, the component exemplifies the project\u0026rsquo;s evolution from a functional tool to a mature, user-centred application.\nSource code and explanations # MultiAliasEditorStrict – Core of Multialias Management (excerpt) # package com.svenruppert.urlshortener.ui.vaadin.components; SNIPP public class MultiAliasEditorStrict extends VerticalLayout { private static final String RX = \u0026#34;^[A-Za-z0-9_-]{3,64}$\u0026#34;; private final Grid\u0026lt;Row\u0026gt; grid = new Grid\u0026lt;\u0026gt;(Row.class, false); private final TextArea bulk = new TextArea(\u0026#34;Aliases (comma/space/newline)\u0026#34;); private final Button insertBtn = new Button(\u0026#34;Take over\u0026#34;); private final Button validateBtn = new Button(\u0026#34;Validate all\u0026#34;); private final String baseUrl; private final Function\u0026lt;String, Boolean\u0026gt; isAliasFree; Server check (true = free) public MultiAliasEditorStrict(String baseUrl, Function\u0026lt;String, Boolean\u0026gt; isAliasFree) { this.baseUrl = baseUrl; this.isAliasFree = isAliasFree; build(); } ==== Public API for the parent view ==== public void validateAll() { var items = new ArrayList\u0026lt;\u0026gt;(grid.getListDataView().getItems().toList()); items.forEach(this::validateRow); grid.getDataProvider().refreshAll(); } public List\u0026lt;String\u0026gt; getValidAliases() { return grid .getListDataView() .getItems() .filter(r -\u0026gt; r.getStatus() == Status.VALID) .map(Row::getAlias) .collect(Collectors.toList()); } public void markSaved(String alias) { setStatus(alias, Status.SAVED, \u0026#34;saved\u0026#34;); } public void markError(String alias, String message) { setStatus(alias, Status.ERROR, (message == null ? \u0026#34;error\u0026#34; : message)); } public long countOpen() { return grid.getListDataView().getItems() .filter(r -\u0026gt; r.getStatus() != Status.SAVED).count(); } public void clearAllRows() { grid.setItems(new ArrayList\u0026lt;\u0026gt;()); } private void setStatus(String alias, Status s, String msg) { grid.getListDataView().getItems().forEach(r -\u0026gt; { if (Objects.equals(r.getAlias(), alias)) { r.setStatus(s); r.setMsg(msg); } }); grid.getDataProvider().refreshAll(); } } Explanation. The component encapsulates all alias management within a standalone UI component. The constructor injects the baseUrl and a server-side availability-check function, keeping the editor testable and decoupling it from specific services. The public API provides the calling view with precise control points: validate all entries, extract valid aliases, set the status of individual rows to \u0026ldquo;SAVED\u0026rdquo; after successful saving, mark error states and count open works. The setStatus method updates the row status and triggers a UI refresh of the Grid DataProvider so that users can see the feedback immediately.\nUIEvent for consistent refreshing # import com.vaadin.flow.component.Component; import com.vaadin.flow.component.ComponentEvent; public class MappingCreatedOrChanged extends ComponentEvent\u0026lt;Component\u0026gt; { public MappingCreatedOrChanged(Component source) { super(source, false); } } Explanation. This minimalist event is the link between the detail dialogue, create view and overview. After successfully saving new aliases, the UI fires this event; the OverviewView registers a listener and reloads its DataProvider. For the user, this means that changes are immediately visible without manual reloading.\nServer-side redirect – consistent methodology control (excerpt) # @Override public void handle(HttpExchange exchange) throws IOException { if (! RequestMethodUtils.requireGet(exchange)) return; // ... further code } Explanation. Recurring checks and error handling are outsourced to utility methods. This reduces dispersion, prevents copy-and-paste divergences and increases the robustness of the routes. This principle is also reflected in the UI: validation, preparation, and persistence are clearly separated and orchestrated through well-defined interfaces.\nNote on the presentation. In this chapter, only the excerpts that are decisive for user guidance are shown. The complete files, as well as the context of the private helper methods (build(), validateRow(\u0026hellip;), bulk input parser, row POJO, and status enum), are located in the feature/advent-2025-day-06 branch. The public API methods access them directly and map the validation-driven interaction flow.\nIntelligent refresh of the overview # The integration of multi-alias management would be incomplete if changes to existing short links were not immediately visible in the overview. This is precisely where intelligent refresh comes in. The user should no longer be required to refresh the view after manually adding or changing aliases. Instead, the application automatically reacts to relevant events and ensures that the displayed data reflects the current state at all times.\nTechnically, this behaviour is implemented via Vaadin\u0026rsquo;s global EventBus. As soon as the MappingCreatedOrChanged event is triggered in the DetailsDialog, the OverviewView registers the signal and internally triggers the update process. The user thus experiences a seamless interaction between editing and overview. The logic responsible for this can be found directly in the constructor of the OverviewView:\nComponentUtil.addListener(UI.getCurrent(), MappingCreatedOrChanged.class, _ -\u0026gt; { logger().info(\u0026#34;Received MappingCreatedOrChanged -\u0026gt; refreshing overview\u0026#34;); refreshPageInfo(); refresh(); }); This code shows how the browser dynamically responds to changes in other UI components. The crucial part is registering the event listener with ComponentUtil.addListener. Every time a MappingCreatedOrChanged event is raised—for example, by the DetailsDialog after saving new aliases—the OverviewView receives it. The refreshPageInfo() and refresh() methods ensure that the displayed data is reloaded. As a result, status indicators, counters, and table contents always remain up to date without requiring user intervention.\nEssential to this implementation is its simplicity and independence. The OverviewView does not know the source of the change. It only reacts to the generic event and fetches the data fresh from the server. This loosely coupled design prevents circular dependencies and makes the system robust to extensions. Future components that also trigger alias changes will be able to use the same event without requiring an adjustment to the overview.\nIt is also noteworthy that the EventBus system in Vaadin enables an apparent decoupling between UI elements. The DetailsDialog and the OverviewView share neither direct references nor mutual knowledge of their respective states. They communicate exclusively about events. This architecture is not only elegant, but also scalable. It enables future expansion of the system with additional listeners, such as notifications, logging, or statistics updates.\nFrom the user\u0026rsquo;s perspective, this results in a reactive, immediate application experience. Changes to aliases or destination addresses are reflected in the overview without any noticeable delay. The user remains in the workflow and is always sure that what he sees reflects the current data state. This seemingly minor adjustment transforms the URL shortener\u0026rsquo;s operation into a modern, responsive interaction model that combines transparency and efficiency.\nImprovements in the Create View # As part of this development step, the Create View also underwent a comprehensive overhaul. The aim was to adapt the process for creating new short URLs more closely to the updated user logic and, at the same time, to standardise workflows. While the detail dialogue is primarily responsible for maintaining existing mappings, the Create View serves as an entry point for new entries, now with the same convenience functions as the edit view.\nThe new version of the Create View offers a more transparent structure and uses a split layout to arrange form elements and preview information side by side. This allows the user to immediately see which aliases they are creating and how they relate to the destination URL. In addition, MultiAliasEditorStrict has been fully integrated, allowing multiple aliases to be captured directly, even when creating a new short URL.\npackage com.svenruppert.urlshortener.ui.vaadin.views; SNIPP all imports import static com.svenruppert.urlshortener.core.DefaultValues.SHORTCODE_BASE_URL; @Route(value = CreateView.PATH, layout = MainLayout.class) public class CreateView extends VerticalLayout implements HasLogger { public static final String PATH = \u0026#34;create\u0026#34;; private static final ZoneId ZONE = ZoneId.systemDefault(); private final URLShortenerClient urlShortenerClient = UrlShortenerClientFactory.newInstance(); private final TextField urlField = new TextField(\u0026#34;Target URL\u0026#34;); private final DatePicker expiresDate = new DatePicker(\u0026#34;Expires (date)\u0026#34;); private final TimePicker expiresTime = new TimePicker(\u0026#34;Expires (time)\u0026#34;); private final Checkbox noExpiry = new Checkbox(\u0026#34;No expiry\u0026#34;); public CreateView() { setSpacing(true); setPadding(true); setSizeFull(); urlField.setWidthFull(); Button saveAllButton = new Button(\u0026#34;Save\u0026#34;); saveAllButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY); Button resetButton = new Button(\u0026#34;Reset\u0026#34;); resetButton.addThemeVariants(ButtonVariant.LUMO_TERTIARY); configureExpiryFields(); FormLayout form = new FormLayout(); form.add(urlField, noExpiry, expiresDate, expiresTime); form.setResponsiveSteps( new FormLayout.ResponsiveStep(\u0026#34;0\u0026#34;, 1), new FormLayout.ResponsiveStep(\u0026#34;900px\u0026#34;, 2) ); form.setColspan(urlField, 2); HorizontalLayout actions = new HorizontalLayout(saveAllButton, resetButton); actions.setWidthFull(); actions.setJustifyContentMode(JustifyContentMode.START); Binder\u0026lt;ShortenRequest\u0026gt; binder = new Binder\u0026lt;\u0026gt;(ShortenRequest.class); binder.forField(urlField) .asRequired(\u0026#34;URL must not be empty\u0026#34;) .withValidator((String url, ValueContext _) -\u0026gt; { var res = UrlValidator.validate(url); return res.valid() ? ValidationResult.ok() : ValidationResult.error(res.message()); }) .bind(ShortenRequest::getUrl, ShortenRequest::setUrl); var editor = new MultiAliasEditorStrict( SHORTCODE_BASE_URL, alias -\u0026gt; { try { return urlShortenerClient.resolveShortcode(alias) == null; } catch (IOException e) { throw new RuntimeException(e); } } ); editor.setSizeFull(); editor.getStyle().set(\u0026#34;padding\u0026#34;, \u0026#34;var(--lumo-space-m)\u0026#34;); saveAllButton.addClickListener(_ -\u0026gt; { var validated = binder.validate(); if (validated.hasErrors()) return; if (!validateExpiryInFuture()) return; if (urlField.getValue() == null || urlField.getValue().isBlank()) { Notification.show(\u0026#34;Target URL is empty\u0026#34;, 2500, Notification.Position.TOP_CENTER); return; } editor.validateAll(); List\u0026lt;String\u0026gt; validAliases = editor.getValidAliases(); if (validAliases.isEmpty()) { Notification.show(\u0026#34;No valid aliases to save\u0026#34;, 2000, Notification.Position.TOP_CENTER); return; } Optional\u0026lt;Instant\u0026gt; expiresAt = computeExpiresAt(); int ok = 0; for (String alias : validAliases) { try { logger().info(\u0026#34;try to save mapping {} / {} \u0026#34;, urlField.getValue(), alias); var customMapping = urlShortenerClient.createCustomMapping(alias, urlField.getValue(), expiresAt.orElse(null)); logger().info(\u0026#34;created customMapping is {}\u0026#34;, customMapping); if (customMapping != null) logger().info(\u0026#34;saved - {}\u0026#34;, customMapping); else logger().info(\u0026#34;save failed for target {} with alias {}\u0026#34;, urlField.getValue(), alias); editor.markSaved(alias); ok++; } catch (Exception ex) { editor.markError(alias, String.valueOf(ex.getMessage())); logger().info(\u0026#34;failed to save URL with alias {}\u0026#34;, alias); } } Notification.show(\u0026#34;Saved: \u0026#34; + ok + \u0026#34; | Open: \u0026#34; + editor.countOpen(), 3500, Notification.Position.TOP_CENTER); }); resetButton.addClickListener(_ -\u0026gt; { clearFormAll(binder); editor.clearAllRows(); }); — SplitLayout var leftCol = new VerticalLayout(new H2(\u0026#34;Create new short links\u0026#34;), form, actions); leftCol.setPadding(false); leftCol.setSpacing(true); leftCol.setSizeFull(); var rightCol = new VerticalLayout(new H2(\u0026#34;Aliases\u0026#34;), editor); rightCol.setPadding(false); rightCol.setSpacing(true); rightCol.setSizeFull(); SplitLayout split = new SplitLayout(leftCol, rightCol); split.setSizeFull(); split.setSplitterPosition(40); add(split); } //... SNIPP private void clearFormAll(Binder\u0026lt;ShortenRequest\u0026gt; binder) { urlField.clear(); noExpiry.clear(); expiresDate.clear(); expiresTime.clear(); binder.setBean(new ShortenRequest()); urlField.setInvalid(false); } } This implementation makes it clear that the Create-View is no longer just a simple input form, but now has the same range of functions as the detail dialogue. The user can capture multiple aliases, validate them directly, and save them with a single click. The integration of the MappingCreatedOrChanged event also ensures that the overview is automatically updated whenever a new mapping is created.\nThe split layout supports a clear visual separation between data entry and alias management. The user remains in the context of their input while also seeing how the selected aliases behave relative to the target URL. The reset function lets you reset inputs without reloading the page.\nThis revision made the Create View an integral part of the consistent user experience. It shares the validation and event logic with the detailed dialogue and thus serves as the foundation for all alias management. The user benefits from consistent behavior in both contexts, while the code remains clearer, more maintainable, and more extensible by reusing key components.\nConsistent validation and error feedback # The growing complexity of the URL shortener\u0026rsquo;s user interface required validation logic that was both reliable and transparent. With the addition of multiple aliases, it became essential to provide precise feedback to the user when inputs are incomplete, duplicated, or syntactically incorrect. This feedback had to appear directly within the context of the respective component to avoid interrupting the interaction flow.\nThe validation system was therefore consistently integrated into Vaadin\u0026rsquo;s existing binder mechanism. The central entry point for this is the Create View, where the target URL field undergoes a strict validation. The implementation uses a dedicated UrlValidator that checks syntactic validity and schema compliance. The following excerpt from the Create View shows the binding and validation of the URL field:\nBinder\u0026lt;ShortenRequest\u0026gt; binder = new Binder\u0026lt;\u0026gt;(ShortenRequest.class); binder.forField(urlField) .asRequired(\u0026#34;URL must not be empty\u0026#34;) .withValidator((String url, ValueContext _) -\u0026gt; { var res = UrlValidator.validate(url); return res.valid() ? ValidationResult.ok() : ValidationResult.error(res.message()); }) .bind(ShortenRequest::getUrl, ShortenRequest::setUrl); This code demonstrates the interaction of mandatory field checking and semantic validation. The validator provides a precise error message for invalid inputs, which is displayed directly below the input field. The user can immediately see why an input was rejected. In this way, Vaadin\u0026rsquo;s classic form validation is supplemented with project-specific logic.\nA similar principle is also used in MultiAliasEditorStrict, which checks each alias entry line by line. In addition to checking syntactic correctness, we also check whether an alias has already been assigned. Visual feedback is provided directly in the alias table, allowing the user to distinguish between valid, saved, and incorrect entries immediately. An excerpt from the corresponding class illustrates this mechanic:\npublic void validateAll() { var items = new ArrayList\u0026lt;\u0026gt;(grid.getListDataView().getItems().toList()); items.forEach(this::validateRow); grid.getDataProvider().refreshAll(); } private void validateRow(Row r) { var a = Objects.requireNonNullElse(r.getAlias(), \u0026#34;\u0026#34;); if (!a.matches(RX)) { r.setStatus(Status.INVALID_FORMAT); r.setMsg(\u0026#34;3–64: A–Z a–z 0–9 - _\u0026#34;); return; } long same = grid.getListDataView().getItems() .filter(x -\u0026gt; x != r \u0026amp;\u0026amp; Objects.equals(a, x.getAlias())) .count(); if (same \u0026gt; 0) { r.setStatus(Status.CONFLICT); r.setMsg(\u0026#34;Duplicate in list\u0026#34;); return; } if (isAliasFree != null) { try { if (!isAliasFree.apply(a)) { r.setStatus(Status.CONFLICT); r.setMsg(\u0026#34;Alias taken\u0026#34;); return; } } catch (Exception ex) { r.setStatus(Status.ERROR); r.setMsg(\u0026#34;Check failed\u0026#34;); return; } } r.setStatus(Status.VALID); r.setMsg(\u0026#34;\u0026#34;); } The validateAll() method iterates over all alias lines and calls an internal check for each. This validation is strict but transparent to the user: each alias line is given its own status indicator and text message. This means that error states are not collected as a list; they are displayed directly within the visual context. This form of inline validation complies with modern UI principles and minimises cognitive interruptions during data entry.\nAnother strength of this architecture is the standardisation of error feedback. Whether it\u0026rsquo;s an invalid URL, an empty alias, or a duplicate name, every discrepancy is handled the same way and communicated through the exact mechanism. This reduces inconsistencies and creates reliable feedback behaviour that users can intuitively understand. At the same time, the code remains easily extensible: new check rules can be added without modifying the existing logic.\nThe interaction between binder validation and alias-related status display thus creates a coherent feedback system. The user immediately learns which inputs have been accepted, which need correction, and when a record has been successfully saved. As a result, validation is no longer perceived as a hurdle, but as an integral part of an accurate and trustworthy user experience.\nCheers Sven\n","date":"11 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-introduction-of-multiple-aliases-part-1/","section":"Posts","summary":"Introduction: More convenience for users # With today’s development milestone for the URL Shortener project, a crucial improvement has been achieved, significantly enhancing the user experience. Up to this point, working with short URLs was functional, but in many ways it was still linear: each destination URL was assigned exactly one alias. This meant users had to create a new short URL for each context or campaign, even when the destination was identical. While this approach was simple, it wasn’t sufficiently flexible to meet real-world application requirements.\n","title":"Advent Calendar 2025 - Introduction of multiple aliases - Part 1","type":"posts"},{"content":" The current UI from the user\u0026rsquo;s point of view # On the first call, the user lands in the overview. The site is built on a Vaadin grid, whose header contains a search bar, paging controls, and a small settings button with a gear icon. The most essential flow begins with the table displaying immediately understandable columns: the shortcode as a clearly typographically separated monospace value with copy action, the original URL as a clickable link, a creation time in local format, and an expiration badge that visually communicates semantic states such as \u0026ldquo;Expired\u0026rdquo;, \u0026ldquo;Today\u0026rdquo; or \u0026ldquo;in n days\u0026rdquo; via theme colours. The whole thing is designed for quick viewing and efficient one-handed operation: a click on a data record opens the detailed dialogue if required; a right-click or the context menu offers direct quick actions; and the gear button can be used to show or hide visible columns live.\nThe source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-05\nCentral to everyday life is the small \u0026ldquo;Settings\u0026rdquo; button on the right of the search bar. It opens the column dialogue and directly affects the grid. This gives the user a lightweight tool to customise the table to their needs without losing context. In the dialogue, the user sees a tidy list of checkboxes—each named after the column header or the internal key. Each click immediately shows or hides the corresponding column; the state is persisted, so that when you revisit it, the view appears as if you had left. The behaviour is deliberately kept minimalist: no \u0026ldquo;Apply\u0026rdquo; button, but immediate feedback, supplemented by an \u0026ldquo;Apply bulk\u0026rdquo; option for collective changes.\nThe table is optimised for common workflows. The shortcode is formatted as a monospace span, and a copy button is right next to it that copies the full short URL to the clipboard. This eliminates the need to highlight text; a quick handover in chat, issue tracker, or email is done with one click. The original URL opens in a new tab, allowing the user to review the landing page without losing track. The creation and expiration times are kept compact; the expiration badge changes colour depending on the remaining term, signalling urgency.\ngrid.addComponentColumn(m -\u0026gt; { var code = new Span(m.shortCode()); code.getStyle().set(\u0026#34;font-family\u0026#34;, \u0026#34;ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace\u0026#34;); var copy = new Button(new Icon(VaadinIcon.COPY)); copy.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE); copy.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Copy ShortUrl\u0026#34;); copy.addClickListener(_ -\u0026gt; { UI.getCurrent().getPage() .executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, SHORTCODE_BASE_URL + m.shortCode()); Notification.show(\u0026#34;Shortcode copied\u0026#34;); }); ... return new HorizontalLayout(code, copy, open, details); }) .setHeader(\u0026#34;Shortcode\u0026#34;) .setKey(\u0026#34;shortcode\u0026#34;) .setAutoWidth(true) .setResizable(true) .setFlexGrow(0); grid.addColumn(m -\u0026gt; DATE_TIME_FMT.format(m.createdAt())) .setHeader(\u0026#34;Created\u0026#34;) .setKey(\u0026#34;created\u0026#34;) .setAutoWidth(true) .setResizable(true) .setSortable(true) .setFlexGrow(0); grid.addComponentColumn(m -\u0026gt; { var pill = new Span(m.expiresAt() .map(ts -\u0026gt; { var days = Duration.between(Instant.now(), ts).toDays(); if (days \u0026lt; 0) return \u0026#34;Expired\u0026#34;; if (days == 0) return \u0026#34;Today\u0026#34;; return \u0026#34;in \u0026#34; + days + \u0026#34; days\u0026#34;; }) .orElse(\u0026#34;No expiry\u0026#34;)); pill.getElement().getThemeList().add(\u0026#34;badge pill small\u0026#34;); ... return pill; }) .setHeader(\u0026#34;Expires\u0026#34;) .setKey(\u0026#34;expires\u0026#34;) .setAutoWidth(true); In addition to the dialogue, there is a second, faster entry point via the context menu. Right-clicking on a line opens actions such as \u0026ldquo;Show details\u0026rdquo;, \u0026ldquo;Open URL\u0026rdquo;, \u0026ldquo;Copy shortcode\u0026rdquo; and \u0026ldquo;Delete\u0026hellip;\u0026rdquo;. This is especially useful if the user already knows what they want to do with the current record without leaving the main view. The flow remains fluid because every action docks directly to the grid.\nGridContextMenu\u0026lt;ShortUrlMapping\u0026gt; menu = new GridContextMenu\u0026lt;\u0026gt;(grid); menu.addItem(\u0026#34;Show details\u0026#34;, e -\u0026gt; e.getItem().ifPresent(this::openDetailsDialog)); menu.addItem(\u0026#34;Open URL\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().open(m.originalUrl(), \u0026#34;_blank\u0026#34;))); menu.addItem(\u0026#34;Copy shortcode\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, m.shortCode()))); menu.addItem(\u0026#34;Delete...\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; confirmDelete(m.shortCode()))); If the user wants to know more or make changes, the application opens a standalone detail dialogue for the selected entry. From an interaction perspective, this is where deeper information and operations converge. The appeal remains deliberately unspectacular and fast:\nprivate void openDetailsDialog(ShortUrlMapping item) { var dlg = new DetailsDialog(urlShortenerClient, item); dlg.addDeleteListener(ev -\u0026gt; confirmDelete(ev.shortCode)); dlg.addOpenListener(ev -\u0026gt; logger().info(\u0026#34;Open URL {}\u0026#34;, ev.originalUrl)); dlg.addCopyShortListener(ev -\u0026gt; logger().info(\u0026#34;Copied shortcode {}\u0026#34;, ev.shortCode)); dlg.addCopyUrlListener(ev -\u0026gt; logger().info(\u0026#34;Copied URL {}\u0026#34;, ev.url)); dlg.addSavedListener(_ -\u0026gt; refresh()); dlg.open(); } As a result, the overview feels like a familiar workboard for the user, which remembers which columns really count, can be copied and opened quickly, and with context menus and a focused detail dialogue offers exactly the depth of interaction that is needed in everyday short URL life — no more, but also no less.\nDetail dialogue on the data record # The detail dialogue is the central interaction element when users want to take a closer look at or edit a single entry in the system. It provides all relevant information about a shortened URL and enables basic operations, such as opening, copying, deleting, or saving the record. All features are designed to be available within the application without context switching.\nThe implementation of the dialogue in the project is in the DetailsDialog class. This class encompasses all the logic for displaying and manipulating a ShortUrlMapping object. The dialogue is built entirely with Vaadin components:\npackage com.svenruppert.urlshortener.ui.vaadin.views.overview; //SNIPP public class DetailsDialog extends Dialog implements HasLogger { private final URLShortenerClient urlShortenerClient; private final ShortUrlMapping mapping; private final DateTimeFormatter DATE_TIME_FMT= DateTimeFormatter.ofPattern(\u0026#34;yyyy-MM-dd HH:mm:ss\u0026#34;) .withZone(ZoneId.systemDefault()); public DetailsDialog(URLShortenerClient urlShortenerClient, ShortUrlMapping mapping) { this.urlShortenerClient = urlShortenerClient; this.mapping = mapping; setHeaderTitle(\u0026#34;Details for \u0026#34; + mapping.shortCode()); var form = new FormLayout(); var urlField = new TextField(\u0026#34;Original URL\u0026#34;); urlField.setValue(Optional.ofNullable(mapping.originalUrl()).orElse(\u0026#34;\u0026#34;)); urlField.setWidthFull(); urlField.setReadOnly(true); var createdAt = new Span(DATE_TIME_FMT.format(mapping.createdAt())); var expiresAt = mapping.expiresAt() .map(ts -\u0026gt; { var days = Duration.between(Instant.now(), ts).toDays(); if (days \u0026lt; 0) return \u0026#34;Expired\u0026#34;; if (days == 0) return \u0026#34;Today\u0026#34;; return \u0026#34;in \u0026#34; + days + \u0026#34; days\u0026#34;; }) .orElse(\u0026#34;No expiry\u0026#34;); var expiresField = new Span(expiresAt); form.addFormItem(urlField, \u0026#34;Original URL\u0026#34;); form.addFormItem(createdAt, \u0026#34;Created\u0026#34;); form.addFormItem(expiresField, \u0026#34;Expires\u0026#34;); var openBtn = new Button(new Icon(VaadinIcon.EXTERNAL_LINK)); openBtn.addThemeVariants(ButtonVariant.LUMO_TERTIARY); openBtn.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Open original URL\u0026#34;); openBtn.addClickListener(_ -\u0026gt; { UI.getCurrent().getPage().open(mapping.originalUrl(), \u0026#34;_blank\u0026#34;); fireEvent(new OpenEvent(this, mapping.originalUrl())); }); var copyShortBtn = new Button(new Icon(VaadinIcon.COPY)); copyShortBtn.addThemeVariants(ButtonVariant.LUMO_TERTIARY); copyShortBtn.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Copy Short URL\u0026#34;); copyShortBtn.addClickListener(_ -\u0026gt; { UI.getCurrent().getPage() .executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, SHORTCODE_BASE_URL + mapping.shortCode()); Notification.show(\u0026#34;Short URL copied\u0026#34;); fireEvent(new CopyShortEvent(this, mapping.shortCode())); }); var copyUrlBtn = new Button(new Icon(VaadinIcon.LINK)); copyUrlBtn.addThemeVariants(ButtonVariant.LUMO_TERTIARY); copyUrlBtn.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Copy Original URL\u0026#34;); copyUrlBtn.addClickListener(_ -\u0026gt; { UI.getCurrent().getPage() .executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, mapping.originalUrl()); Notification.show(\u0026#34;Original URL copied\u0026#34;); fireEvent(new CopyUrlEvent(this, mapping.originalUrl())); }); var deleteBtn = new Button(new Icon(VaadinIcon.TRASH)); deleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR, ButtonVariant.LUMO_TERTIARY); deleteBtn.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Delete Short URL\u0026#34;); deleteBtn.addClickListener(_ -\u0026gt; { fireEvent(new DeleteEvent(this, mapping.shortCode())); }); var saveBtn = new Button(\u0026#34;Save\u0026#34;, _ -\u0026gt; { urlShortenerClient.update(mapping); fireEvent(new SavedEvent(this)); close(); }); var buttons = new HorizontalLayout(openBtn, copyShortBtn, copyUrlBtn, deleteBtn, saveBtn); add(form, buttons); } public Registration addOpenListener(ComponentEventListener\u0026lt;OpenEvent\u0026gt; listener) { return addListener(OpenEvent.class, listener); } public Registration addCopyShortListener(ComponentEventListener\u0026lt;CopyShortEvent\u0026gt; listener) { return addListener(CopyShortEvent.class, listener); } public Registration addCopyUrlListener(ComponentEventListener\u0026lt;CopyUrlEvent\u0026gt; listener) { return addListener(CopyUrlEvent.class, listener); } public Registration addDeleteListener(ComponentEventListener\u0026lt;DeleteEvent\u0026gt; listener) { return addListener(DeleteEvent.class, listener); } public Registration addSavedListener(ComponentEventListener\u0026lt;SavedEvent\u0026gt; listener) { return addListener(SavedEvent.class, listener); } public static class OpenEvent extends ComponentEvent\u0026lt;DetailsDialog\u0026gt; { private final String originalUrl; public OpenEvent(DetailsDialog src, String originalUrl) { super(src, false); this.originalUrl = originalUrl; } public String originalUrl() { return originalUrl; } } public static class CopyShortEvent extends ComponentEvent\u0026lt;DetailsDialog\u0026gt; { private final String shortCode; public CopyShortEvent(DetailsDialog src, String shortCode) { super(src, false); this.shortCode = shortCode; } public String shortCode() { return shortCode; } } public static class CopyUrlEvent extends ComponentEvent\u0026lt;DetailsDialog\u0026gt; { private final String url; public CopyUrlEvent(DetailsDialog src, String url) { super(src, false); this.url = url; } public String url() { return url; } } public static class DeleteEvent extends ComponentEvent\u0026lt;DetailsDialog\u0026gt; { private final String shortCode; public DeleteEvent(DetailsDialog src, String shortCode) { super(src, false); this.shortCode = shortCode; } public String shortCode() { return shortCode; } } public static class SavedEvent extends ComponentEvent\u0026lt;DetailsDialog\u0026gt; { public SavedEvent(DetailsDialog src) { super(src, false); } } } The architecture follows a clear pattern: the dialogue displays data but does not trigger a direct UI update. Instead, it fires specific events (ComponentEvents) that are caught by the calling View (OverviewView). This means that the dialogue remains independent of its environment – a concept that can be implemented particularly elegantly in Vaadin.\nThe meaning of the individual buttons is immediately understandable: the open button opens the browser with the original URL, the two copy buttons copy the respective address to the clipboard, and the delete button sends a delete event that is further processed in the overview. It is particularly noteworthy that the save button internally calls urlShortenerClient.update(), thereby synchronising changes directly through the existing client-server infrastructure.\nThis makes the detail dialogue a self-contained UI element that encapsulates both presentation and interaction without dragging business logic into the interface. This pattern promotes reusability, testability, and a clear separation of responsibilities.\nContext menu of grid rows # In the project, the context menu is anchored to the OverviewView and bound to the ShortUrlMapping objects‘ grid. The source code shows how this menu is structured and what actions are offered in it:``\nGridContextMenu\u0026lt;ShortUrlMapping\u0026gt; menu = new GridContextMenu\u0026lt;\u0026gt;(grid); menu.addItem(\u0026#34;Show details\u0026#34;, e -\u0026gt; e.getItem().ifPresent(this::openDetailsDialog)); menu.addItem(\u0026#34;Open URL\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().open(m.originalUrl(), \u0026#34;_blank\u0026#34;))); menu.addItem(\u0026#34;Copy shortcode\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, m.shortCode()))); menu.addItem(\u0026#34;Delete...\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; confirmDelete(m.shortCode()))); The logic is simple but effective: each menu item performs a clearly defined action. These actions draw directly on the existing mechanisms of the OverviewView – such as the openDetailsDialog() for detail views or confirmDelete() for removing a data record.\nEach menu action uses the method e.getItem().ifPresent(...)to ensure that a context object (that is, a selected entry in the grid) exists. This pattern protects the application from null references and makes the menu robust against unforeseen UI states.\nThe integration into the OverviewView is straightforward and follows Vaadin\u0026rsquo;s component-oriented approach. By binding to the grid, the menu is automatically linked to the respective rows. Users can thus interact with the data in a context-sensitive manner – an ergonomic advantage over classic toolbar or button solutions.\nThe connection to the detail dialogue is seamless: If the user selects \u0026ldquo;Show details\u0026rdquo; in the context menu, the DetailsDialog for the corresponding ShortUrlMapping opens immediately. The interaction between the grid, context menu, and dialogue remains completely encapsulated within the view, keeping the code easy to maintain and understand.\nEvent integration between Overview and DetailDialog # The interaction between the OverviewView and the DetailsDialog is the functional heart of the editing workflow. It combines a tabular overview with a detailed view of individual datasets and ensures that changes in the dialogue are immediately reflected in the grid. The concept follows Vaadin\u0026rsquo;s component-oriented event mechanism, in which UI components can trigger their own events, which are processed by other elements – in this case, the OverviewView .\nIn the OverviewView, the dialogue opens when the user selects \u0026ldquo;Show details\u0026rdquo; from the context menu or double-clicks an entry. The openDetailsDialog() method takes over this task and, at the same time, binds all relevant listeners to the dialogue:\nprivate void openDetailsDialog(ShortUrlMapping item) { var dlg = new DetailsDialog(urlShortenerClient, item); dlg.addDeleteListener(ev -\u0026gt; confirmDelete(ev.shortCode)); dlg.addOpenListener(ev -\u0026gt; logger().info(\u0026#34;Open URL {}\u0026#34;, ev.originalUrl)); dlg.addCopyShortListener(ev -\u0026gt; logger().info(\u0026#34;Copied shortcode {}\u0026#34;, ev.shortCode)); dlg.addCopyUrlListener(ev -\u0026gt; logger().info(\u0026#34;Copied URL {}\u0026#34;, ev.url)); dlg.addSavedListener(_ -\u0026gt; refresh()); dlg.open(); } Each listener processes a specific event triggered by the dialogue itself. The trick is that the DetailsDialog defines these events as their own, type-safe subclasses of ComponentEvent. This allows the OverviewView to respond to actions in a targeted manner without knowing the dialogue\u0026rsquo;s implementation details. Examples include DeleteEvent, CopyUrlEvent, CopyShortEvent, OpenEvent, and SavedEvent.\nThe crucial point is the refresh() at the end of the listener chain. This ensures that after a change to the dialogue, the grid data is reloaded. This keeps the display consistent and immediately reflects the current persistence state.\nThe dialogue itself triggers these events as soon as user actions occur. In the event of a deletion action, this is done via:\ndeleteBtn.addClickListener(_ -\u0026gt; { fireEvent(new DeleteEvent(this, mapping.shortCode())); }); Or when copying the short URL:\ncopyShortBtn.addClickListener(_ -\u0026gt; { UI.getCurrent().getPage() .executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, SHORTCODE_BASE_URL + mapping.shortCode()); Notification.show(\u0026#34;Short URL copied\u0026#34;); fireEvent(new CopyShortEvent(this, mapping.shortCode())); }); By triggering its own events, the dialogue can operate decoupled from its environment. It doesn\u0026rsquo;t know who calls it or what happens to the data afterwards – it just reports that an action occurred. This decoupling is an essential aspect of good UI architecture: it enables reuse and testable components without side effects.\nThe OverviewView then takes responsibility for updating the system in response to these events. Exquisite is the use of ComponentEventListener, which has type safety and clean demarcation of the event classes. Vaadin thus offers a clearly structured and idiomatic way to model complex UI flows.\nAnother example shows the connection between the deletion process and the server communication. When a DeleteEvent is fired, the OverviewView calls the internal method confirmDelete() to display a security prompt and, upon confirmation, delete the record server-side. Only then does the grid refresh again, so the removed entry disappears immediately.\nThis event-based integration ensures a consistent and reactive processing flow. Users experience changes in real time, without manual updates. For developers, the model offers a clear separation of responsibilities: the DetailsDialog encapsulates presentation and interaction, while OverviewView orchestrates and synchronises them. This creates a UI structure that can be flexibly expanded and easily supplemented with new actions.\nValidation and error feedback in the dialogue # A central element of the edit dialogue is input validation. This is to ensure that only correct and complete data is transmitted to the server. The focus is on checking URLs, as they form the core of every ShortUrlMapping.\nIn the current project, validation is handled using the Vaadin Binder. The binder connects the UI fields to the properties of the underlying data model and provides built-in support for validations and error feedback.\nA typical excerpt from the archive shows how this validation is implemented in practice:\nbinder.forField(urlField) .asRequired(\u0026#34;URL must not be empty\u0026#34;) .withValidator(url -\u0026gt; { var validate = UrlValidator.validate(url); return validate; }, \u0026#34;Only HTTP(S) URLs allowed\u0026#34;) .bind(ShortUrlMapping::originalUrl, null); Several steps are combined here:\nRequired field check – asRequired() ensures that the field is not left empty. If no input is made, Vaadin automatically displays an error message. Content validation : withValidator() is also used to validate the URL format. This uses the UrlValidator helper class. The UrlValidator itself is a standalone utility class that syntactically checks whether a string is a valid HTTP or HTTPS URL. The implementation is:\npackage com.svenruppert.urlshortener.core.urlmapping; import java.net.URI; public final class UrlValidator { private UrlValidator() {} public static boolean validate(String url) { if (url == null || url.isBlank()) { return false; } try { var uri = URI.create(url); var scheme = uri.getScheme(); return scheme != null \u0026amp;\u0026amp; (scheme.equalsIgnoreCase(\u0026#34;http\u0026#34;) || scheme.equalsIgnoreCase(\u0026#34;https\u0026#34;)); } catch (IllegalArgumentException e) { return false; } } } This implementation is deliberately kept simple and uses only standard Java means. It verifies that the URL has a valid schema and starts with HTTP or HTTPS. This excludes all other protocols – an important aspect to avoid potential security risks (e.g. file:// or javascript://).\nThe result of this validation is immediately displayed in the UI. If the user enters an invalid URL, an error message will appear below the input field that comes from the message set withValidator() (\u0026ldquo;Only HTTP(S) URLs allowed\u0026rdquo;). The Vaadin binder automatically controls this visual feedback.\nThe interaction among Binder, TextField, and UrlValidator ensures consistent, user-friendly, and secure input validation. In addition, the system can be expanded if necessary, for example, to include additional checks (e.g. URL accessibility, blacklist filters or regex-based structure tests) without changing the existing architecture.\nThis validation ensures that the DetailsDialog is not just a display element, but also a reliable filter that catches erroneous input before it enters the persistence layer. This reduces error scenarios, increases data quality, and significantly improves the user experience.\nUX fine-tuning and conclusion # The user experience (UX) of the interaction among the OverviewView, the DetailsDialog, and supporting UI components, such as the ColumnVisibilityDialog, is the result of numerous targeted design decisions. The goal was to create a reactive yet simple interface that avoids visual overload while still offering complete control over the data.\nThe application consistently follows Vaadin\u0026rsquo;s guiding principles for a component-based UI. All interactions – from column selection to detail view – are encapsulated in clearly defined classes. The user benefits from an intuitive interaction logic: every action is available where it makes semantic sense, and every change is immediately visible.\nAn example of this UX approach is direct feedback on user actions. When a URL is copied or opened, a notification appears immediately:\nNotification.show(\u0026#34;Shortcode copied\u0026#34;); This micro-feedback improves the perception of the system response and confirms to the user that their action was successful. The same principle applies to several components, for example, when copying the original URL or saving it in the DetailsDialog.\nAnother part of the fine-tuning is the deliberate use of Vaadin theme variants to differentiate controls visually. For example, the delete button in the detail dialogue clearly signals its critical function through the combination of LUMO_ERRORandLUMO_TERTIARY:\ndeleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR, ButtonVariant.LUMO_TERTIARY); This colouring adheres to the established Lumo design system and ensures that dangerous actions are immediately recognisable, without requiring additional explanatory text.\nThe modalization of the detail dialogue also contributes to user guidance:\nsetModal(true); setDraggable(true); setResizable(true); This keeps the user\u0026rsquo;s focus clearly on the task at hand, while dragging and resizing offer flexibility. Especially in data-rich grids, this allows the dialogue to adapt to the situation without leaving the application.\nAnother UX aspect is maintaining the work context. After each action – such as saving or deleting – the grid is reloaded by the refresh() method without losing filter or paging information. This was deliberately kept this way in the implementation:\nvar req = new UrlMappingListRequest(searchField.getValue(), pagination.getCurrentPage(), pagination.getPageSize()); var mappings = urlShortenerClient.list(req); grid.setItems(mappings); This means the user remains in the same position within the data view and maintains their orientation and workflow rhythm.\nThe OverviewView\u0026rsquo;s structure demonstrates how a consistent user experience can be achieved through small, targeted design decisions. Instead of complex UI frameworks, the application uses Vaadin\u0026rsquo;s native component toolkit and combines it with a clear event architecture and lightweight HTTP communication.\nResult # The implementation of the editing workflow via the DetailsDialog shows how a modern, reactive application flow can be created with pure Vaadin Flow and Core Java– without external frameworks, reflection, or client-side JavaScript logic. The key lies in clean component decoupling, precise event control, and immediate user feedback.\nThe user receives a UI that adapts to their workflows: fast, comprehensible and error-resistant. At the same time, developers can create a maintainable architecture with clear responsibilities: the view orchestrates, the dialogue interacts, and the client synchronises. This interplay of simplicity and clarity is the actual UX fine-tuning of this implementation – and at the same time, the basis for the further expansion stages of the project.\nCheers Sven\n","date":"10 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-from-grid-to-detail-understanding-the-user-experience-in-the-short-url-manager/","section":"Posts","summary":"The current UI from the user’s point of view # On the first call, the user lands in the overview. The site is built on a Vaadin grid, whose header contains a search bar, paging controls, and a small settings button with a gear icon. The most essential flow begins with the table displaying immediately understandable columns: the shortcode as a clearly typographically separated monospace value with copy action, the original URL as a clickable link, a creation time in local format, and an expiration badge that visually communicates semantic states such as “Expired”, “Today” or “in n days” via theme colours. The whole thing is designed for quick viewing and efficient one-handed operation: a click on a data record opens the detailed dialogue if required; a right-click or the context menu offers direct quick actions; and the gear button can be used to show or hide visible columns live.\n","title":"Advent Calendar - 2025 - From Grid to Detail: Understanding the User Experience in the Short-URL Manager","type":"posts"},{"content":" Server-Side Extension: PreferencesHandler and REST Interfaces # The server-side extension for dynamic column visibility follows the same design logic as the UI: simplicity, clear accountability, and a precise data flow. While the OverviewView and the ColumnVisibilityDialog form the interface for the interaction, several specialized REST handlers handle the processing and persistence of the user settings. Their job is to process incoming JSON requests, validate them, translate them into domain operations, and return or store the current state.\nThe source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-04\nHere\u0026rsquo;s the screenshot of the version we\u0026rsquo;re implementing now.\nThe central link is the PreferencesStore interface, which defines both read and write operations for column preferences. It is divided into two functional aspects – loading and updating:\npublic interface PreferencesStore extends PreferencesLookup, PreferencesUpdater, HasLogger { } public interface PreferencesLookup { Map\u0026lt;String, Boolean\u0026gt; load(String userId, String viewId); } public interface PreferencesUpdater { void saveColumnVisibilities(String userId, String viewId, Map\u0026lt;String, Boolean\u0026gt; visibility); void delete(String userId, String viewId, String columnKey); void delete(String userId, String viewId); void delete(String userId); } This structure forms the foundation for the subsequent merchant classes. Each REST endpoint includes a clearly defined operation—loading, editing, or deleting column preferences. This creates a granular API that is idempotent on the one hand and enables precise error handling on the other.\nThe first port of call is the ColumnVisibilityHandler, which accepts POST and DELETE requests. It is responsible for loading all of a user\u0026rsquo;s column settings within a view. The handler first checks that the request is complete, and then returns a flat JSON map that maps column names and visibility states:\n@Override public void handle(HttpExchange ex) throws IOException { switch(ex.getRequestMethod()) { case \u0026#34;POST\u0026#34; -\u0026gt; handleLoad(ex); case \u0026#34;DELETE\u0026#34; -\u0026gt; handleDeleteAll(ex); case \u0026#34;OPTIONS\u0026#34; -\u0026gt; allow(ex, \u0026#34;POST, DELETE, OPTIONS\u0026#34;); default -\u0026gt; methodNotAllowed(ex, \u0026#34;POST, DELETE, OPTIONS\u0026#34;); } } private void handleLoad(HttpExchange ex) throws IOException { var req = fromJson(readBody(ex.getRequestBody()), ColumnInfoRequest.class); var vis = store.load(req.userId(), req.viewId()); writeJson(ex, OK, toJson(vis == null ? Map.of() : vis)); } In addition, there are two specialized variants that separate the editing operations from each other. The ColumnVisibilitySingleHandler handles PUT and DELETE requests for individual columns. It takes a JSON object with user ID, view ID, and column name and updates the persistence accordingly:\nprivate void handleSingleEdit(HttpExchange ex) throws IOException { var req = fromJson(readBody(ex.getRequestBody()), ColumnSingleEditRequest.class); var visibility = Map.of(req.columnKey(), req.visible()); store.saveColumnVisibilities(req.userId(), req.viewId(), visibility); writeJson(ex, OK, toJson(Map.of(\u0026#34;status\u0026#34;, \u0026#34;ok\u0026#34;))); } For bulk uploads, the ColumnVisibilityBulkHandler is used. This variant allows you to update multiple column states in a single request at the same time. The concept of bulk processing reduces latency and minimizes server-side write operations, for example, if the user changes many checkboxes in the dialog:\nprivate void handleBulkEdit(HttpExchange ex) throws IOException { var req = fromJson(readBody(ex.getRequestBody()), ColumnEditRequest.class); store.saveColumnVisibilities(req.userId(), req.viewId(), req.changes()); writeJson(ex, OK, toJson(Map.of(\u0026#34;status\u0026#34;, \u0026#34;ok\u0026#34;))); } All three handlers follow the same design pattern: they validate the request, call the appropriate methods of the PreferencesStore and send back the correct HTTP status code. If successful, the server will respond with 200 OK or 204 No Content, and if the input is incorrect, it will respond with 400 Bad Request. This convention makes the interface robust and easy to test.\nThe new API thus forms the backbone of the personalization logic. At the same time, it is an example of the loose coupling between UI and persistence: the client knows nothing about the storage strategy, the server nothing about the representation. Both communicate via well-defined JSON structures. This principle keeps the application extensible and independent of concrete implementation details – be it an in-memory store or long-term storage in EclipseStore.\nThe PreferencesClient: Round trip between UI and server # The connection between the user interface and persistence is established by the PreferencesClient. This component takes on the task of addressing the REST endpoints of the server side via HTTP, serializing data and converting the results into structures that can be used within the Vaadin UI. The client thus acts as an intermediary between the interaction logic of the interface and the data storage of the server – a classic link between display and state.\nThe structure of the PreferencesClient follows the established pattern of the project\u0026rsquo;s other service clients. It uses the Java standard library with HttpClient, HttpRequest and HttpResponse and completely dispenses with external dependencies. Communication takes place via clearly defined endpoints, all of which are documented in the class:\n/** * Client for server-side column visibility preferences. * * Endpoints: * - POST /admin/preferences/columns -\u0026gt; load * - DELETE /admin/preferences/columns -\u0026gt; delete all (for a view) * - PUT /admin/preferences/columns/edit -\u0026gt; bulk edit * - PUT /admin/preferences/columns/single -\u0026gt; single edit */ public final class ColumnVisibilityClient implements HasLogger { The client provides a set of methods that directly correspond to the server-side REST endpoints. The typical flow of a round trip starts with loading the stored visibility information:\npublic Map\u0026lt;String, Boolean\u0026gt; load(String userId, String viewId) throws IOException, InterruptedException { var reqDto = new ColumnInfoRequest(userId, viewId); var req = requestBuilder(PATH_ADMIN_PREFERENCES_COLUMNS) . POST(jsonBody(reqDto)) .build(); var resp = http.send(req, HttpResponse.BodyHandlers.ofString(UTF_8)); if (resp.statusCode() == 200) { var body = resp.body(); if (body == null || body.isBlank()) return Collections.emptyMap(); var parsed = parseJson(body); return parsed.entrySet().stream() .collect(Collectors.toMap(Map.Entry::getKey, e -\u0026gt; Boolean.parseBoolean(e.getValue()))); } if (resp.statusCode() == 204) return Collections.emptyMap(); throw new IOException(\u0026#34;Unexpected HTTP \u0026#34; + resp.statusCode() + \u0026#34; while loading column visibilities: \u0026#34; + resp.body()); } The method generates a simple JSON object from the user and view IDs and sends it to the server as a POST request. The answer contains a map of column names to truth values that can be used directly in the UI. Missing values are interpreted as true , which is in line with the principle of full visibility.\nThe client offers two variants for changes: the editing of individual columns and the bulk update. While the editSingle method specifically adjusts a column state, editBulk allows you to commit multiple changes in a single request. This separation corresponds to the semantic structure of the REST handlers:\npublic void editSingle(String userId, String viewId, String columnKey, boolean visible) throws IOException, InterruptedException { var reqDto = new ColumnSingleEditRequest(userId, viewId, columnKey, visible); var req = requestBuilder(PATH_ADMIN_PREFERENCES_COLUMNS_SINGLE) . PUT(jsonBody(reqDto)) .build(); var resp = http.send(req, HttpResponse.BodyHandlers.ofString(UTF_8)); if (resp.statusCode() != 200) { throw new IOException(\u0026#34;Unexpected HTTP \u0026#34; + resp.statusCode() + \u0026#34; on single edit: \u0026#34; + resp.body()); } } This method illustrates the principle of idempotency: multiple calls with the same data result in the same state without producing side effects. This is especially important for reactive user interfaces that update states asynchronously.\nIn combination with the ColumnVisibilityService, which acts as a wrapper, this creates a clear communication flow:\nThe UI interacts with the service via the ColumnVisibilityDialog. The service calls the appropriate endpoints via the PreferencesClient. The server side validates, stores and returns current states. The service reflects the new values in the grid. The entire system is therefore deterministic, comprehensible and fault-tolerant. If a transfer fails, the previous state is retained and the user can initiate the operation again. The PreferencesClient thus forms the technical basis for the stable persistence of visual preferences – a simple but extremely robust bridge between interaction and storage.\nPersistence and EclipseStore integration # The storage of user preferences for column visibility is done in the same persistence system that was already introduced in the previous parts of the project: EclipseStore. This decision not only follows the consistency of the architecture design, but also underlines the goal of bringing together all system states – whether functional or visual – in a coherent data model. The persistence of user preferences thus becomes an equal part of the application.\nThe implementation of the PreferencesStore interface plays a central role in this. There are several concrete variants in the architecture: an in-memory version for tests and volatile scenarios as well as an EclipseStore-supported version for productive operation. Both follow the same contract model, but differ in the underlying storage medium.\nThe following excerpt shows the interface structure on which the implementations are based:\ncom.svenruppert.urlshortener.api.store.preferences.PreferencesStore public interface PreferencesStore extends PreferencesLookup, PreferencesUpdater, HasLogger { } com.svenruppert.urlshortener.api.store.preferences.PreferencesLookup public interface PreferencesLookup { Map\u0026lt;String, Boolean\u0026gt; load(String userId, String viewId); } com.svenruppert.urlshortener.api.store.preferences.PreferencesUpdater public interface PreferencesUpdater { void saveColumnVisibilities(String userId, String viewId, Map\u0026lt;String, Boolean\u0026gt; visibility); void delete(String userId, String viewId, String columnKey); void delete(String userId, String viewId); void delete(String userId); } These clear contracts form the basis for various storage strategies. In particular, the EclipseStore variant (EclipsePreferencesStore) integrates seamlessly into the existing object graph. When saving, it checks whether an entry for the combination of user and view ID already exists and updates the visibility information accordingly. A new entry is only created if no previous state exists. This logic ensures idempotency and prevents unnecessary duplicates.\nThe following fragment from the EclipseStore implementation illustrates the principle:\ncom.svenruppert.urlshortener.api.store.provider.eclipsestore.patitions.EclipsePreferencesStore (simplified excerpt) @Override public void saveColumnVisibilities(String userId, String viewId, Map\u0026lt;String, Boolean\u0026gt; visibility) { var userPrefs = dataRoot.preferences().computeIfAbsent(userId, _ -\u0026gt; new HashMap\u0026lt;\u0026gt;()); var viewPrefs = userPrefs.computeIfAbsent(viewId, _ -\u0026gt; new HashMap\u0026lt;\u0026gt;()); viewPrefs.putAll(visibility); storage.storeRoot(dataRoot); } This method illustrates the simplicity and directness of persistence: All preferences are stored in the root object, so they automatically become part of the transactional storage in the EclipseStore. The entire system benefits from object persistence without classic database tables or ORM layers.\nA key feature of this solution is its resilience. Because EclipseStore automatically synchronizes changes, user preferences are reliably maintained even if the system terminates unexpectedly. In addition, the application benefits from the direct object addressing that EclipseStore offers: no complicated ORM mappings are necessary, and changes to the data structure are automatically applied. Storage thus remains as natural as the modeling itself.\nIn addition, the persistence layer remains extensible. By separating the interface and implementation, alternative storage mechanisms can also be used in the future – such as an encrypted file variant for particularly sensitive environments or a network-based solution for multi-user systems. The existing code of the UI and the server handler would not have to be adapted for this, as all access takes place via the PreferencesStore interface.\nWith this integration, the persistence path is complete: from the user to the dialog, the service, the client, the REST interfaces and finally to the EclipseStore, the data flow is consistently typed and consistent. This finally makes column visibility a part of the permanent system state – a visible sign that user interaction and data storage are no longer separate worlds in this architecture, but two perspectives on the same, persistent context.\nArchitecture and Event Flow # The introduction of dynamic column visibility has not only spawned new UI and server components, but has also refined the entirety of the application architecture. It has created a consistent flow of events that aligns all levels, from the user interface down to persistence. This consistency makes the function not only robust, but also expandable and testable.\nAt the center of this flow is the OverviewView as the trigger for user interaction. When the user presses the gear icon, the ColumnVisibilityDialog opens, which in turn communicates via the ColumnVisibilityService. This service calls the ColumnVisibilityClient, which in turn calls the server\u0026rsquo;s REST endpoints. The server processes the request via the appropriate handlers – such as ColumnVisibilityHandler, ColumnVisibilitySingleHandler or ColumnVisibilityBulkHandler – and writes the changes to the EclipseStore via the PreferencesStore. Finally, the return channel ensures that the stored visibility is automatically restored during the next initialization.\nThe sequence can be schematically represented as follows:\nUsers → OverviewView → ColumnVisibilityDialog → ColumnVisibilityService → ColumnVisibilityClient → REST API → PreferencesHandler → PreferencesStore → EclipseStore → Persistent Storage Each of these components fulfils a clearly defined role and communicates exclusively via well-defined interfaces. In this way, the architecture strictly follows the principle of separation of concerns. The UI part remains completely decoupled from persistence, while the server logic does not require any knowledge of the frontend. Changes to one layer do not have an immediate effect on the other layers.\nA particularly elegant aspect can be seen in the interaction between the user interface and the event system. The OverviewView is connected to the StoreEvents event system . As soon as changes to the database are detected on the server side, a signal is sent to the UI via a publish/subscribe pattern. The application responds to this with a synchronized refresh:\nsubscription = StoreEvents.subscribe(_ -\u0026gt; getUI().ifPresent(ui -\u0026gt; ui.access(this::refresh))); This mechanic, which is already known from the previous days of the Advent calendar, ensures that the user interface and saved state always match. It forms the foundation of reactivity throughout the project.\nThe event flow within the application is thus bidirectional: On the one hand, the user initiates interactions that lead through the round trip to persistence. On the other hand, persistent changes can in turn have an effect on the UI. This interplay between action and reaction creates a dynamic coherence that makes the system tangibly alive.\nCompared to classic web applications, in which the state between client and server is often fragmented or only temporary, this architecture establishes a continuous data cycle. Every step is traceable and typified – from the checkbox in the dialog to the saved Boolean map in the EclipseStore. This minimizes sources of error and significantly increases the maintainability of the application.\nAll in all, it can be seen that the addition of dynamic column visibility is not just a new function, but a maturation of the entire system architecture. It combines UI, event control and persistence into a coherent whole and thus lays the foundation for further personalization mechanisms based on the same principles.\nUX and ergonomics: Self-determined work # With the introduction of dynamic column visibility, the relationship between user and application is fundamentally changing. While the previous functions were primarily aimed at consistency, stability and clarity, one aspect is now coming to the fore that was previously mainly implicit: the self-determination of the user. The OverviewView interface is no longer a rigid reflection of developers\u0026rsquo; decisions, but becomes a customizable tool that is subordinate to the individual ways of working of its users.\nThis change is already evident in the way the dialogue has been shaped. The ColumnVisibilityDialog follows the principle of minimal friction: the user should be able to adjust his view without losing context or feeling cognitive load. Therefore, the dialog opens modally, focuses on the current task and offers only those controls that are relevant for the decision. Each checkbox represents a column – nothing more, nothing less. The immediate feedback in the grid after each change ensures that the user experiences the effects of their decision directly.\nThe combination of immediate feedback and persistent storage has a psychologically important effect: it creates trust. When a system visibly responds to the user\u0026rsquo;s preferences and restores them unchanged the next time it is opened, the feeling of stability and control is created. This is one of the cornerstones of good interaction design – the system should not only work, but also appear reliable.\nThe placement of the function on the surface also follows ergonomic considerations. The gear icon that opens the dialog is deliberately designed to be unobtrusive. It is accessible at all times, but not dominant. This keeps the focus on the content while keeping control over the presentation always within reach. The user decides for himself when and to what extent he makes adjustments. The system does not impose itself, but reacts.\nCombined with the automatic restoration of preferences at launch, it creates a gentle personalization that feels organic. The application not only remembers states – it also remembers habits. If you have hidden certain columns, you signal a personal usage pattern that will be quietly respected on your next visit. This form of implicit personalization is efficient because it does not require additional dialogues, profile settings or user accounts and still creates an individual experience.\nThe developer perspective is not left out. The architecture was designed in such a way that the expansion of further personal settings – such as sort orders, column widths or layout preferences – would be possible seamlessly. The existing concept of the PreferencesStore allows the inclusion of additional parameters without structural changes. This creates an extensible system that may seem simple in the user experience, but is highly modular in the background.\nThis makes the dynamic column visibility feature more than just a convenience feature. It is an expression of a philosophy: the user should be able to find himself in the application. Software that adapts instead of demanding adaptation creates acceptance and productivity at the same time. The user remains the focus, technology takes a back seat – exactly where it has its greatest effect.\nTechnical reflection and safety aspects # The introduction of dynamic column visibility is not only a functional extension, but also a technical statement. It shows that personalization in a modern web application does not have to be at odds with robustness, security, and maintainability. Rather, individualization can be integrated as a controlled and comprehensible extension of the existing security model.\nFrom a technical point of view, the new function touches on several security-relevant levels. First of all, the handling of user IDs should be mentioned. In the current implementation, the username \u0026ldquo;admin\u0026rdquo; serves as a placeholder, but in a production environment, this information comes from an authenticated context. It is important that the preferences are strictly tied to the authenticated user in order to ensure a clear delimitation of the visibility areas. A user may only access their own stored preferences – never anyone else\u0026rsquo;s.\nThe interfaces themselves are designed in such a way that they already implicitly assume this separation. Each request contains both a userId and a viewId. The REST handlers check that both values are present and valid. Missing or empty fields will result in a controlled rejection of the request:\nif (isBlank(req.userId()) || isBlank(req.viewId())) { writeJson(ex, BAD_REQUEST, \u0026#34;userId and viewId required\u0026#34;); return; } This simple but effective validation protects against non-specific calls and prevents unwanted manipulation of the database. In combination with the REST semantics, which only allow idempotent operations (POST, PUT, DELETE), it ensures that the application behaves deterministically even under heavy load or during repeated requests.\nAnother important aspect concerns data integrity within the PreferencesStore. Since the stored values are organized in a map\u0026lt;string, Boolean\u0026gt; the question arises as to how to deal with invalid or manipulated data. The implementation uses type checking and defensive programming for this purpose: Only known column names are adopted, unknown or incorrect keys are discarded. This keeps the persisted state consistent, even if a failed client sends data.\nvar knownKeys = grid.getColumns() .stream() .map(Grid.Column::getKey) .filter(Objects::nonNull) .toList(); Map\u0026lt;String, Boolean\u0026gt; state = service.mergeWithDefaults(knownKeys); This logic in the UI prevents unauthorized or inappropriate values from being stored in the first place. The dialog works exclusively with those columns that actually exist in the grid, and is therefore inherently resistant to manipulation attempts on the client side.\nIn addition to functional safety, resilience to errors also plays an essential role. All communication paths between client and server are embedded in try-catch blocks. If the connection is lost or the server does not respond, the application will continue to be usable. The service reports errors via logging without blocking user interaction:\ntry { client.editBulk(userId, viewId, changes); } catch (IOException | InterruptedException e) { logger().warn(\u0026#34;Persist bulk failed {}: {}\u0026#34;, changes.keySet(), e.toString()); } This approach ensures that server communication failures do not have a negative impact on the user experience. The user can continue working; its changes are automatically synchronized on the next successful connection.\nOverall, it can be seen that the security concept of the application has not been weakened by the expansion, but strengthened. The integrity of the system is maintained through clear interfaces, typed data structures, defensive programming and strict separation of user contexts. The dynamic column visibility proves that user freedom and system security are not mutually exclusive, but – if implemented carefully – reinforce each other.\nResult # With the introduction of dynamic column visibility, the development of the administration interface has reached a decisive milestone. What began as a static, system-centric overview in the first days of the Advent calendar has now developed into an interactive, user-centered platform. The application learns, reacts and remembers – it becomes a tool that adapts to the user instead of forcing him to get used to its structures.\nInstead of replacing existing components, they have been expanded and precisely linked to each other. The new ColumnVisibilityDialog fits seamlessly into the existing UI concept, the PreferencesClient uses the same mechanisms as the previous REST clients, and persistence via the EclipseStore closes the circle of dataflows. Everything interlocks, without breaks or special paths. This homogeneity is not a coincidence, but the result of an architecture that has anchored openness and coherence as fundamental principles from the very beginning.\nFrom a user experience perspective, the extension is a step towards self-determination. Users can now personalize their workspace without having to struggle through menus and options. The interface remains light, the behavior is comprehensible and the stored preferences create trust in the stability of the application. The system reacts to user decisions instead of forcing them – a paradigm shift that noticeably increases the quality of interaction.\nThe clear separation between user context, visibility logic and data storage prevents manipulation and ensures data integrity. The use of typed maps and well-defined REST endpoints minimizes sources of error. Persistence via EclipseStore also offers the advantage that the application remains consistent at all times, even in the event of abrupt interruptions or partial failures.\nLooking ahead opens up new possibilities. On the basis of the now established preference system, further personalization dimensions can be implemented: column sequences, sorting preferences, color schemes or layout options. By loosely coupling UI, service and persistence, the system is ready for these extensions without having to change existing structures. Integration with user profiles or role-based access models could also be built on it.\nCheers Sven\n","date":"9 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-columnvisibilitydialog-part-2/","section":"Posts","summary":"Server-Side Extension: PreferencesHandler and REST Interfaces # The server-side extension for dynamic column visibility follows the same design logic as the UI: simplicity, clear accountability, and a precise data flow. While the OverviewView and the ColumnVisibilityDialog form the interface for the interaction, several specialized REST handlers handle the processing and persistence of the user settings. Their job is to process incoming JSON requests, validate them, translate them into domain operations, and return or store the current state.\n","title":"Advent Calendar - 2025 - ColumnVisibilityDialog - Part 2","type":"posts"},{"content":" From observer to designer: User control at a glance # With the fourth day of the Advent calendar, the perspective on the application changes fundamentally. While in the previous expansion stages users reacted primarily to prepared structures, an element of active design is now being added. The \u0026ldquo;Overview\u0026rdquo; has so far been a central mirror of the system state: It showed all saved short links, allowed them to be filtered and provided an in-depth insight into individual objects with the detailed dialog. But the structure and visibility of the columns were immutable up to this point. The user always saw the selection that the developer had defined as meaningful. With the introduction of dynamic column visibility, this state is removed.\nFrom observer to designer: User control at a glance Concept: Visibility as a user preference The new UI interaction: ColumnVisibilityDialog Integration with the OverviewView The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-04\nHere\u0026rsquo;s the screenshot of the version we\u0026rsquo;re implementing now.\nThe grid in the OverviewView is thus transformed from a static display element into a customizable tool. Users can choose which columns are visible and which are hidden. This step seems minor at first glance, but it leads to a new form of interaction between user and application. The system no longer determines the surface structure, but reacts to individual preferences. The application enters into a dialogue with the user, in which the view of data becomes part of the personalization.\nThis extension is a direct continuation of the basis laid in the previous days. The filter and search mechanisms from the first part have taken control of the data flow. The integration of EclipseStore has made the state visible and at the same time established permanent storage in the system. Finally, the detail dialog created the separation between the overview and the object context. With this in mind, the new feature is the next logical step: it shifts control over the presentation of data from the system to the user without complicating the underlying technical structure.\nFrom a user experience perspective, this change is significant. Those who work with the administration interface on a daily basis develop their own rhythm in dealing with data. Some users focus on expiring links, others on manual aliases, or on the traffic of their short links. For all these scenarios, a static column arrangement is a hindrance. The ability to hide irrelevant information and tailor the view to your own work processes creates not only convenience, but also efficiency. The user thus turns from a passive observer into an active designer of his or her work environment.\nThe implementation of this principle in Vaadin is based on a conscious balance between simplicity and persistence. On the one hand, the function should be immediately available: One click opens a dialog, a handful of checkboxes controls the visibility of the columns, and the result becomes immediately visible. On the other hand, these attitudes should not remain fleeting. The selected configuration is permanently saved and automatically restored on each reboot. This persistence in visual behavior takes up the idea of the visible system state and transfers it to the user level.\nThe introduction of dynamic column visibility thus marks the beginning of a new chapter in the development of the administration interface: It is no longer just about the presentation of data, but about the design of one\u0026rsquo;s own work context. This feature marks a paradigm shift in the UI design of the project. Instead of predefined structures, a flexible, learning interface is created that adapts to individual ways of working and thus takes a decisive step towards true user-centricity.\nConcept: Visibility as a user preference # The introduction of dynamic column visibility not only marks an ergonomic improvement in the user interface, but also a conceptual expansion of the application architecture. For the first time, an aspect of user interaction is permanently personalized – regardless of the session or system state. This innovation requires a separate model for preferences that works consistently on both the client and server side. Visibility thus becomes a stored property that is mediated between the user and the application.\nThe central idea is that each view in the application has its own identity. For the OverviewView, this means that its configuration – such as which columns are visible – is clearly stored under a view ID. At the same time, each user instance is identified, for example by means of a user ID, and treated as a carrier of individual preferences. These two dimensions – user and view – are key to storing visibility information. The concept thus follows a clear scheme: a user ID, a view ID and an assignment of column names to truth values.\nThe introduction of this logic required a new interface between the UI and the backend. The application had to learn to deal with semantic information about user decisions, not just data objects. On the server side, this is done by the new PreferencesHandler, which serves as a REST endpoint and allows both loading and saving the settings. On the client side, the new PreferencesClient component communicates with this handler and manages the transfer of the data structures. The connecting format is deliberately simple: a map\u0026lt;string, Boolean\u0026gt; in which each key corresponds to the column name and the value describes the visibility. This clarity makes it possible to keep the mechanism generic and later to extend it to other areas of the application.\nThe decision to model preferences as a separate entity is an important architectural step. It separates application data—such as short links and expiration information—from the presentation data that drives the user\u0026rsquo;s behavior within the interface. In this way, the persistence layer is expanded by a new category: no longer only technical but also contextual states. This pattern is reminiscent of the separation between domain logic and UI logic, which is central to the clean architecture approach, and consistently transfers it to the behavior of the user interface.\nAnother goal of the implementation is transparency. The preferences should not act as a black box, but remain traceable at all times. Changes to the column configuration are therefore immediately visible and saved at the same time. The user does not feel a break between the interaction and the result, and yet their behavior is preserved permanently. This dual nature – volatile response and persistent storage – makes the system a learning tool. The application remembers how its users work and automatically reproduces this behavior on the next call.\nThis adds a semantic layer to the UI concept: visibility is no longer understood as a technical property of the grid component, but as an expression of an individual way of working. The application itself becomes a customizable medium that adapts to people, not the other way around. In this perspective, the dynamics of column visibility become a symbol for the transition from standardized operation to personalized control – a principle that will also be reflected in the technical implementation in the following chapters.\nThe new UI interaction: ColumnVisibilityDialog # With the introduction of dynamic column visibility, the application\u0026rsquo;s user interface gets a new level of interaction. The previously immutable structure of the OverviewView is extended by a component that allows the user to directly influence the structure of his working environment. At the heart of this is the ColumnVisibilityDialog – a dialog box that lists all available columns in the overview table and controls their visibility via simple checkboxes. It embodies the idea of a configurable but intuitive interface.\nThe concept of dialogue follows a deliberate reduction: no cluttered settings menu, no hidden options, but a clear, focused place for a single task. When the dialog is opened, the application reads the saved preferences of the current user via the PreferencesClient and sets the checkboxes according to the stored values. Each checkbox represents a column of the OverviewView – such as \u0026ldquo;Shortcode\u0026rdquo;, \u0026ldquo;URL\u0026rdquo;, \u0026ldquo;Created\u0026rdquo; or \u0026ldquo;Expires\u0026rdquo;. Changes are immediately visible: As soon as the user selects or deselect a checkbox, the corresponding column in the grid is shown or hidden. This creates a direct interplay between action and reaction, which strengthens the perception of control.\nThe ColumnVisibilityDialog is deliberately implemented as an independent component. It extends the Vaadin class Dialog and follows the same architectural philosophy as the already known DetailsDialog from the previous days. This parallel allows it to fit seamlessly into the existing structure: it has a clear lifecycle logic, its own events and a minimalist layout that is designed for comprehensibility and efficiency. The dialog typically contains a vertical list of checkboxes that is dynamically generated from the grid\u0026rsquo;s column configuration. A save button permanently updates preferences through the PreferencesClient, while a \u0026ldquo;Reset\u0026rdquo; button restores the original default view.\nFrom a technical point of view, the dialog is based on a simple but elegant data flow. When opened, the client loads a map\u0026lt;string, Boolean\u0026gt; whose keys correspond to the column identifiers. This map is converted into UI elements, with the current visibility mirrored directly in the grid. Clicking on \u0026ldquo;Save\u0026rdquo; triggers the method saveColumnVisibilities(userId, viewId, visibilityMap), which transmits the changed settings to the server. The JSON format is deliberately left flat to be both human-readable and easily testable. Server-side storage in EclipseStore ensures that these changes are not only retained for the current session, but also permanently.\nThe interplay of dialog, grid and client shows the strength of the modular approach. The ColumnVisibilityDialog does not know any details about persistence or grid implementation – it works exclusively via clearly defined interfaces. This keeps the component independent, reusable, and easily expandable. This pattern is particularly important in the Vaadin architecture because it ensures the loose coupling between user interaction and system health.\nFrom a user experience perspective, the dialog performs two tasks at the same time. It provides control without complexity and creates confidence in the stability of the system. Users experience that their individual decisions are not only temporary, but are structurally anchored in the application. Every adjustment is reproducible, every change is reversible. This creates a natural balance between freedom and security, which reflects the character of the entire project. The ColumnVisibilityDialog thus becomes a visible expression of the idea that an administration interface should not only provide tools, but also adapt to the way its users work.\nTo illustrate this, the following is an original excerpt from the dialog, which shows the coupling between checkboxes and grid columns as well as the collection of changes for an optional collective persistence. The code works directly on the column keys of the grid and reflects state changes back to the interface without any detours.\nMap\u0026lt;String, Boolean\u0026gt; pending = new LinkedHashMap\u0026lt;\u0026gt;(); grid.getColumns().forEach(col -\u0026gt; { final String key = col.getKey(); if (key == null) return; only addressable columns boolean visible = state.getOrDefault(key, true); col.setVisible(visible); var cb = new Checkbox(col.getHeaderText() != null ? col.getHeaderText() : key, visible); cb.addValueChangeListener(ev -\u0026gt; { boolean v = Boolean.TRUE.equals(ev.getValue()); col.setVisible(v); pending.put(key, v); }); form.add(cb); }); var btnApply = new Button(\u0026#34;Apply bulk\u0026#34;, _ -\u0026gt; { if (!pending.isEmpty()) { service.setBulk(new LinkedHashMap\u0026lt;\u0026gt;(pending)); pending.clear(); } close(); }); btnApply.addThemeVariants(ButtonVariant.LUMO_PRIMARY); The dialog deliberately delegates the actual persistence to the service. This node abstracts the transport and writes the changed visibilities to the server in one go. The following fragment shows the corresponding method in the service, which encapsulates the round trip and at the same time remains fault-tolerant.\npublic void setBulk(Map\u0026lt;String, Boolean\u0026gt; changes) { if (changes == null || changes.isEmpty()) return; try { client.editBulk(userId, viewId, changes); } catch (IOException | InterruptedException e) { logger().warn(\u0026#34;Persist bulk failed {}: {}\u0026#34;, changes.keySet(), e.toString()); } } In this way, the ColumnVisibilityDialog remains lean and focused on the interaction, while the persistence logic is handled centrally via the service and the underlying client. This division of responsibilities ensures that the user experience feels clear and responsive without compromising the technical integrity of the application.\nIntegration with the OverviewView # The introduction of the ColumnVisibilityDialog would have remained incomplete if it had not been organically embedded in the existing OverviewView. This is exactly where the strength of the architecture becomes apparent, which was designed for loose coupling and clear responsibilities from the very beginning. The OverviewView serves as the application\u0026rsquo;s central window—connecting data, filters, paging, and interaction. Now it is expanding this spectrum to include a persistent personalization dimension.\nAt the center of the integration is an inconspicuous but meaningful button in the search bar: the gear icon. This icon acts as an entry point into the configuration logic. The decision not to hide the dialog in a menu or in a separate view follows the idea of control available at all times. Users should be able to customize the presentation of the data exactly where they look at it. The code snippet from the OverviewView illustrates this direct integration:\nbtnSettings.addThemeVariants(ButtonVariant.LUMO_TERTIARY); btnSettings.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Column visibility\u0026#34;); btnSettings.addClickListener(_ -\u0026gt; new ColumnVisibilityDialog\u0026lt;\u0026gt;(grid, columnVisibilityService).open()); With just a few lines, the entire functionality is connected. The button opens a new instance of the dialog, which receives the current grid and the corresponding service instance. This service instance manages all operations on the stored preferences. In this way, a clear data flow is created: the OverviewView calls the dialog, the dialog interacts with the ColumnVisibilityService, and the ColumnVisibilityService in turn communicates with the server via the ColumnVisibilityClient.\nThe initialization of the service takes place when the view is attached to the UI. This phase is ideal for loading user contexts and creating initial states. The relevant code block from the onAttach method shows this flow:\n@Override protected void onAttach(AttachEvent attachEvent) { super.onAttach(attachEvent); this.columnVisibilityService = new ColumnVisibilityService(columnVisibilityClient, \u0026#34;admin\u0026#34;, \u0026#34;overview\u0026#34;); var keys = grid.getColumns().stream() .map(Grid.Column::getKey) .filter(Objects::nonNull) .toList(); var vis = columnVisibilityService.mergeWithDefaults(keys); grid.getColumns().forEach(c -\u0026gt; { var k = c.getKey(); if (k != null) c.setVisible(vis.getOrDefault(k, true)); }); refresh(); subscription = StoreEvents.subscribe(_ -\u0026gt; getUI().ifPresent(ui -\u0026gt; ui.access(this::refresh))); } When setting up the view, the service is first created and linked to the user and view ID. All known column names are then determined and their visibility is coordinated with the stored preferences. Each column will then be shown or hidden according to its saved setting. This way, the column states are not fleeting, but consistent between sessions and restarts.\nThis integration is deliberately implemented discreetly. The dialog is invoked only when the user becomes active, and the service automatically takes into account the saved settings. In this way, the application is not overloaded with new interactions, but its behavior is organically expanded by a context-aware memory.\nOne aspect of this architecture is the way it maintains the balance between control and simplicity. The user does not need to load or save any configuration – everything happens in the background. The view remains responsive, the server records the state, and the dialog acts as an interface between the two. This invisible connection between the surface and the persistence layer gives the application the lightness that keeps it intuitive and efficient despite the growing variety of functions.\nCheers Sven\n","date":"8 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-columnvisibilitydialog-part-1/","section":"Posts","summary":"From observer to designer: User control at a glance # With the fourth day of the Advent calendar, the perspective on the application changes fundamentally. While in the previous expansion stages users reacted primarily to prepared structures, an element of active design is now being added. The “Overview” has so far been a central mirror of the system state: It showed all saved short links, allowed them to be filtered and provided an in-depth insight into individual objects with the detailed dialog. But the structure and visibility of the columns were immutable up to this point. The user always saw the selection that the developer had defined as meaningful. With the introduction of dynamic column visibility, this state is removed.\n","title":"Advent Calendar - 2025 - ColumnVisibilityDialog - Part 1","type":"posts"},{"content":" Client contract from a UI perspective # In this project, the user interface not only serves as a graphical layer on top of the backend, but is also part of the overall contract between the user, the client, and the server. This part focuses on the data flow from the UI\u0026rsquo;s perspective: how inputs are translated into structured requests, how the client forwards them, and what feedback the user interface processes.\nClient contract from a UI perspective Client layer (URLShortenerClient): Extensions Server API and Handler Persistence and Store Implementations InMemory Implementation EclipseStore Implementation Uniformity and Compatibility Advantages of the approach Domain Model and Defaults ShortUrlMapping as a central link DefaultValues – Central System Constants AliasPolicy with Logging JSON serialisation and deserialization Extending Serialisation in JsonUtils Extending Deserialization Purge of incoming JSON data Optimisation of JSON output Consistency and interoperability Security and robustness in the UI flow Input validations in the generation dialogue Defensive navigation in the detail dialogue Dealing with the Clipboard API Consistent feedback and fault tolerance Result The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-03\nHere\u0026rsquo;s the screenshot of the version we\u0026rsquo;re implementing now.\nThe basis of the contract is the class ShortenRequest, which was extended in this development step with the new field expiresAt. This field serves as a central repository for expiration dates and is entirely optional – meaning existing clients will continue to function even without this attribute. The UI client is thus both backwards-compatible and future-proof.\npublic class ShortenRequest { private String url; private String shortURL; private Instant expiresAt; public ShortenRequest(String url, String shortURL, Instant expiresAt) { this.url = url; this.shortURL = shortURL; this.expiresAt = expiresAt; } public Instant getExpiresAt() { return expiresAt; } public void setExpiresAt(Instant expiresAt) { this.expiresAt = expiresAt; } } The CreateView passes this object to the URLShortenerClient, which handles communication with the server. The decisive factor here is that the user interface does not establish HTTP connections directly but delegates this task to a dedicated client component. This keeps the UI lean and testable, while the client is centrally responsible for logging, error handling and request generation. The central interface is the createCustomMapping method, which maps the extended contract:\npublic ShortUrlMapping createCustomMapping(String alias, String url, Instant expiredAt) throws IOException { logger().info(\u0026#34;Create custom mapping alias=\u0026#39;{}\u0026#39; url=\u0026#39;{}\u0026#39; expiredAt=\u0026#39;{}\u0026#39;\u0026#34;, alias, url, expiredAt); URL shortenUrl = serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL(); HttpURLConnection connection = (HttpURLConnection) shortenUrl.openConnection(); connection.setRequestMethod(\u0026#34;POST\u0026#34;); connection.setDoOutput(true); connection.setRequestProperty(CONTENT_TYPE, JSON_CONTENT_TYPE); var shortenRequest = new ShortenRequest(url, alias, expiredAt); String body = shortenRequest.toJson(); try (OutputStream os = connection.getOutputStream()) { os.write(body.getBytes(UTF_8)); } int status = connection.getResponseCode(); if (status == 200 || status == 201) { try (InputStream is = connection.getInputStream()) { String jsonResponse = new String(is.readAllBytes(), UTF_8); ShortUrlMapping shortUrlMapping = fromJson(jsonResponse, ShortUrlMapping.class); return shortUrlMapping; } } if (status == 409) { throw new IllegalArgumentException(\u0026#34;Alias already in use\u0026#34;); } throw new IOException(\u0026#34;Unexpected status: \u0026#34; + status); } This example illustrates how precisely the UI and the client are coordinated. The UI passes a fully populated domain instance (ShortenRequest) that contains all the required fields. The client handles the serialisation, performs the communication, and returns a ShortUrlMapping in response. The UI then displays the relevant data immediately.\nA central design principle in this interaction is \u0026ldquo;data instead of commands\u0026rdquo;. The UI does not send specific control commands, but always sends complete data objects. The backend decides how to operate based on these objects. This decoupling has several advantages:\nExtensibility: New fields (e.g. expiresAt) can be added without breaking existing APIs. Traceability: Every operation is fully traceable via the Request object. Security: The client can validate inputs before converting them into HTTP requests. In the UI, the client\u0026rsquo;s responses are used to provide feedback and update the current view. Not only is the shortcode displayed, but the entire mapping object is also displayed, which already contains all the values calculated on the server side. The user interface thus always remains in line with the actual system state.\nThis pattern – a clear contract structure between UI, client and server – creates stability and maintainability in the long term. Changes to the backend API do not require adjustments to the UI logic, as long as the underlying data contract remains unchanged. This establishes a binding communication path that enables technical evolution without impairing user interaction.\nClient layer (URLShortenerClient): Extensions # The client layer forms the technical bridge between the user interface and the server. It translates data objects into HTTP requests, monitors the communication, and transfers the results back into domain objects. This chapter shows how the existing class URLShortenerClient has been extended to support the new expiration concept (expiresAt) while ensuring clean, valid communication with the server.\nThe starting point was the existing function createCustomMapping(String alias, String url), which an overloaded variant has now supplemented. This accepts an additional expiration date (Instant expiredAt) and performs all necessary steps to transfer the data to the server in a complete and compliant manner.\npublic ShortUrlMapping createCustomMapping(String alias, String url, Instant expiredAt) throws IOException { logger().info(\u0026#34;Create custom mapping alias=\u0026#39;{}\u0026#39; url=\u0026#39;{}\u0026#39; expiredAt=\u0026#39;{}\u0026#39;\u0026#34;, alias, url, expiredAt); if (alias != null \u0026amp;\u0026amp; !alias.isBlank()) { var validate = AliasPolicy.validate(alias); if (!validate.valid()) { var reason = validate.reason(); throw new IllegalArgumentException(reason.defaultMessage); } } URL shortenUrl = serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL(); HttpURLConnection connection = (HttpURLConnection) shortenUrl.openConnection(); connection.setRequestMethod(\u0026#34;POST\u0026#34;); connection.setDoOutput(true); connection.setRequestProperty(CONTENT_TYPE, JSON_CONTENT_TYPE); var shortenRequest = new ShortenRequest(url, alias, expiredAt); String body = shortenRequest.toJson(); logger().info(\u0026#34;createCustomMapping - body - \u0026#39;{}\u0026#39;\u0026#34;, body); try (OutputStream os = connection.getOutputStream()) { os.write(body.getBytes(UTF_8)); } int status = connection.getResponseCode(); if (status == 200 || status == 201) { try (InputStream is = connection.getInputStream()) { String jsonResponse = new String(is.readAllBytes(), UTF_8); logger().info(\u0026#34;createCustomMapping - jsonResponse - {}\u0026#34;, jsonResponse); ShortUrlMapping shortUrlMapping = fromJson(jsonResponse, ShortUrlMapping.class); logger().info(\u0026#34;shortUrlMapping .. {}\u0026#34;, shortUrlMapping); return shortUrlMapping; } } if (status == 409) throw new IllegalArgumentException(\u0026#34;Alias already in use\u0026#34;); } throw new IOException(\u0026#34;Unexpected status: \u0026#34; + status); } With this method, the extended expiration date is seamlessly integrated into the existing communication flow. The UI calls this method via CreateView, keeping the extension transparent to the user – the new functionality is immediately available without changing the user experience.\nIn addition to improving the method signature, the consistency of HTTP communication has also been improved. Instead of manually setting headers, the constant JSON_CONTENT_TYPE is now used consistently to avoid format errors and ensure unique typing. This standardisation reduces the risk of inconsistent requests and facilitates later protocol extensions (e.g., for authenticating or signing requests).\nAnother critical point is logging. The URLShortenerClient logs all relevant steps – from request creation to response processing. This transparency is crucial for understanding the exact process in the event of an error. Especially during the development phase and when integrating new features such as expiresAt, logging provides valuable insights into the timing, format and status of the data transfer.\nA typical log snippet might look like this:\nINFO Create custom mapping alias=\u0026#39;test123\u0026#39; url=\u0026#39;https://example.com\u0026#39; expiredAt=\u0026#39;2025-12-31T23:59:00Z\u0026#39; INFO createCustomMapping - body - \u0026#39;{\u0026#34;url\u0026#34;: \u0026#34;https://example.com\u0026#34;, \u0026#34;alias\u0026#34;: \u0026#34;test123\u0026#34;, \u0026#34;expiresAt\u0026#34;: \u0026#34;2025-12-31T23:59:00Z\u0026#34;}\u0026#39; INFO createCustomMapping - jsonResponse - \u0026#39;{\u0026#34;shortCode\u0026#34;: \u0026#34;ex-9A7\u0026#34;, \u0026#34;originalUrl\u0026#34;: \u0026#34;https://example.com\u0026#34;, \u0026#34;expiresAt\u0026#34;: \u0026#34;2025-12-31T23:59:00Z\u0026#34;}\u0026#39; INFO shortUrlMapping.. ShortUrlMapping[shortCode=ex-9A7, url=https://example.com, expiresAt=2025-12-31T23:59:00Z] This structured logging follows a clear pattern that makes all the steps involved traceable. It is intended not only for debugging, but also for integration into an audit or monitoring solution in the long term.\nIn conclusion, it should be noted that the client layer is now fully context-sensitive. It recognises optionally passed expiration dates, validates inputs, communicates clearly structured JSON payloads, and correctly interprets server-side responses. This creates a robust, well-defined interface between the user interface and the backend that can easily accommodate future extensions, such as custom policies or metadata.\nServer API and Handler # The server layer forms the backbone of the entire architecture. It receives requests, interprets the data structures, and coordinates the creation, storage, or deletion of short links. With the introduction of the expiration date (expiresAt), the API has been extended by a significant semantic dimension. The goal was to integrate this new information into the existing request-response flow without causing compatibility issues with older clients.\nThe central point of contact for incoming POST requests to create a short link is the ShortenHandler. This handles the JSON payload, performs validations, and interacts with the UrlMappingStore. In doing so, the processing has been extended to correctly extract the expiration date from the JSON object and pass it to persistence.\nfinal String body = readBody(ex.getRequestBody()); ShortenRequest req = fromJson(body, ShortenRequest.class); if (isNullOrBlank(req.getUrl())) { writeJson(ex, BAD_REQUEST, \u0026#34;Missing \u0026#39;url\u0026#39;\u0026#34;); return; } final Result\u0026lt;ShortUrlMapping\u0026gt; urlMappingResult = store.createMapping(req.getShortURL(), req.getUrl(), req.getExpiresAt()); urlMappingResult .ifPresentOrElse(success -\u0026gt; logger().info(\u0026#34;mapping created success {}\u0026#34;, success.toString()), failed -\u0026gt; logger().info(\u0026#34;mapping created failed - {}\u0026#34;, failed)); urlMappingResult .ifSuccess(mapping -\u0026gt; { final Headers h = ex.getResponseHeaders(); h.add(\u0026#34;Location\u0026#34;, \u0026#34;/r/\u0026#34; + mapping.shortCode()); writeJson(ex, fromCode(201), toJson(mapping)); }) .ifFailure(errorJson -\u0026gt; { try { var parsed = JsonUtils.parseJson(errorJson); var errorCode = Integer.parseInt(parsed.get(\u0026#34;code\u0026#34;)); var message = parsed.get(\u0026#34;message\u0026#34;); writeJson(ex, fromCode(errorCode), message); } catch (Exception e) { writeJson(ex, CONFLICT, errorJson); } }); The ShortenHandler deliberately does not use a framework, but works with native Java APIs (HttpExchange, HttpURLConnection) to keep the control flow completely transparent. This decision is not only for traceability but also enables a precise understanding of how HTTP works at the lowest level. The code clearly shows the steps of the server cycle: read body, deserialise, validate, invoke domain logic, and write response.\nAnother example is the ListHandler, which has been slightly modified to take advantage of the modern Sequenced API (List.getFirst()), thus increasing readability:\nprivate static String first(Map\u0026lt;String, List\u0026lt;String\u0026gt;\u0026gt; q, String key) { var v = q.get(key); return (v == null || v.isEmpty()) ? null : v.getFirst(); } Special attention is paid to robust JSON data processing. Here, it has been ensured that line breaks and escape sequences do not cause parsing problems. In the JsonUtils module, a cleanup step was introduced before parsing:\ns = s.replaceAll(\u0026quot;\\\\n\u0026quot;, \u0026ldquo;\u0026rdquo;);\nThis prevents multi-line JSON data from leading to errors – a typical stumbling block for APIs that process manually generated or logged payloads.\nOverall, the server API deliberately remains flat and declarative. Each operation represents a clearly defined domain action; there is no excessive branching or hidden state change. By adding expiresAt, this style is retained, and the system continues to react deterministically: A request creates a mapping, optionally with an expiration date, and returns the complete record as JSON.\nThis simplicity is not a coincidence, but an expression of a design principle that runs through all levels of the application: explicit data flows instead of implicit magic. The result is a system that remains understandable for both users and developers and can be reliably expanded.\nPersistence and Store Implementations # The persistence layer is the foundation on which the entire system\u0026rsquo;s reliability rests. With the introduction of the expiration date (expiresAt), it had to be expanded accordingly so that this new information can be reliably stored, queried and evaluated – regardless of whether the data is stored in memory or in a persistent database such as EclipseStore.\nAt the center of this layer is the UrlMappingUpdater interface, which has been extended by a new method. This method adds the Instant expiredAt parameter to the previous signatures, so that the persistence layer can now explicitly handle expiration times.\npublic interface UrlMappingUpdater { Result\u0026lt;ShortUrlMapping\u0026gt; createMapping(String originalUrl); Result\u0026lt;ShortUrlMapping\u0026gt; createMapping(String alias, String url); Result\u0026lt;ShortUrlMapping\u0026gt; createMapping(String alias, String url, Instant expiredAt); boolean delete(String shortCode); } This clearly states that every implementation must process flow information. This adaptation follows the principle of contract-based design – the interface defines which capabilities the specific implementation must possess without prescribing their technical details.\nInMemory Implementation # The first customization was done in the InMemoryUrlMappingStore. This class is primarily used for tests and volatile runtime environments and stores all mappings in a ConcurrentHashMap. By extending the createMapping methods, expiration data is now correctly transferred to the MappingCreator.\n@Override public Result\u0026lt;ShortUrlMapping\u0026gt; createMapping(String alias, String originalUrl, Instant expiredAt) { logger().info(\u0026#34;alias: {} - originalUrl: {} - expiredAt: {} \u0026#34;, alias, originalUrl, expiredAt); return creator.create(alias, originalUrl, expiredAt); } In the MappingCreator itself, the expiration time is integrated into the ShortUrlMapping and stored directly when created:\npublic Result\u0026lt;ShortUrlMapping\u0026gt; create(String alias, String url, Instant expiredAt) { logger().info(\u0026#34;createMapping - alias=\u0026#39;{}\u0026#39; / url=\u0026#39;{}\u0026#39; / expiredAt=\u0026#39;{}\u0026#39;\u0026#34;, alias, url, expiredAt); final String shortCode; if (!isNullOrBlank(alias)) { if (repository.containsKey(alias)) { return Result.failure(\u0026#34;Alias already exists\u0026#34;); } shortCode = alias; } else { shortCode = generator.generate(); } var mapping = new ShortUrlMapping(shortCode, url, Instant.now(clock), Optional.ofNullable(expiredAt)); store.accept(mapping); return Result.success(mapping); } This pattern shows that expiration information, if any, is directly part of the ShortUrlMapping domain object. The code is deliberately simple: no additional state, no special treatment, but only an optional value.\nEclipseStore Implementation # The EclipseStoreUrlMappingStore has also been adapted for permanent storage. The same principle applies here, but with a focus on long-term persistence.\n@Override public Result\u0026lt;ShortUrlMapping\u0026gt; createMapping(String alias, String originalUrl, Instant expiredAt) { logger().info(\u0026#34;alias: {} - originalUrl: {} - expiredAt: {}\u0026#34;, alias, originalUrl, expiredAt); return creator.create(alias, originalUrl, expiredAt); } Tightly coupling with the same MappingCreator ensures complete consistency in behaviour between the InMemory and EclipseStore variants. The only difference is the storage duration: While the data in the InMemory store is lost on restart, it remains persistent in the EclipseStore.\nUniformity and Compatibility # A central goal of these adaptations was the complete equal treatment of all persistent species. Whether it\u0026rsquo;s testing, development mode, or production, all paths use the same objects and methods. This eliminates the risk of divergent logic across different storage types. Changes to the domain, such as the introduction of expiresAt, therefore only have to be implemented in one place.\nAdvantages of the approach # This consistent standardization brings several advantages:\nTransparency: Every mapping operation is traceable and documented in the log. Consistency: InMemory and EclipseStore stores behave identically. Extensibility: New storage mechanisms (e.g., SQL, key-value store, cloud) can be easily added as long as they fulfil the interface contract. With this extension, the persistence layer becomes the system\u0026rsquo;s reliable backbone. The expiration date is now a full-fledged part of the data model – precisely recorded, securely stored and retrievable at any time. This means that future functions such as automatic cleaning of expired entries or time-based statistics can be implemented directly on this basis.\nDomain Model and Defaults # The class ShortenRequest has been extended to include the field expiresAt. This field allows you to pass an optional expiration time, which is set by the user in the UI and delivered to the server as an instant in JSON format. This information thus becomes a full-fledged component of the data model and can be further processed at both the transport layer and the persistence layer.\npublic class ShortenRequest { private String url; private String shortURL; private Instant expiresAt; public ShortenRequest(String url, String shortURL, Instant expiresAt) { this.url = url; this.shortURL = shortURL; this.expiresAt = expiresAt; } public Instant getExpiresAt() { return expiresAt; } public void setExpiresAt(Instant expiresAt) { this.expiresAt = expiresAt; } public String toJson() { var a = shortURL == null ? \u0026#34;\\\u0026#34;null\\\u0026#34;\u0026#34; : \u0026#34;\\\u0026#34;\u0026#34; + JsonUtils.escape(shortURL) + \u0026#34;\\\u0026#34;\u0026#34;; var b = expiresAt == null ? \u0026#34;\\\u0026#34;null\\\u0026#34;\u0026#34; : \u0026#34;\\\u0026#34;\u0026#34; + JsonUtils.escape(expiresAt.toString()) + \u0026#34;\\\u0026#34;\u0026#34;; return \u0026#34;\u0026#34;\u0026#34; { \\\u0026#34;url\\\u0026#34;: \\\u0026#34;%s\\\u0026#34;, \\\u0026#34;alias\\\u0026#34;: %s, \\\u0026#34;expiresAt\\\u0026#34;: %s } \u0026#34;\u0026#34;\u0026#34;.formatted(JsonUtils.escape(url), a, b); } } It is important to note that the expiresAt value is not mandatory. This keeps existing clients, and server calls compatible, even if they don\u0026rsquo;t set the field. The domain model was deliberately designed to remain backwards-compatible and extensible, an essential principle when introducing new features.\nShortUrlMapping as a central link # The ShortUrlMapping class represents the central data element between the client and the server. It contains all the relevant information of a short link: the generated shortcode, the destination URL, the creation date and, optionally, the expiration date. By using Optional, the possible absence of an expiration date is explicitly modelled.\npublic record ShortUrlMapping(String shortCode, String originalUrl, Instant createdAt, Optional expiresAt) { }\nThis decision underscores the model\u0026rsquo;s functional properties: an immutable data element that is fully defined only when it is created. Changes to a mapping are always made through new creations or explicit updates – never through silent mutations.\nDefaultValues – Central System Constants # In addition to the model classes, the DefaultValues class has also been extended. It contains constants that are used throughout the application, especially the base URL for generated short links.\npublic final class DefaultValues { TODO - must be editable by user public static final String SHORTCODE_BASE_URL = \u0026#34;https://3g3.eu/\u0026#34;; public static final int ADMIN_SERVER_PORT = 9090; public static final String ADMIN_SERVER_HOST = \u0026#34;localhost\u0026#34;; public static final String ADMIN_SERVER_PROTOCOL = \u0026#34;http\u0026#34;; more path definitions ... } The constant SHORTCODE_BASE_URL serves as the basic component for generating and displaying short links in the user interface. Although it is currently statically defined, it has already been noted that it will be dynamically configurable in a future iteration. This lays the foundation for flexible deployment scenarios in which different environments (e.g., development, test, production) can use their own base URLs.\nAliasPolicy with Logging # Another component of the domain model is the AliasPolicy, which defines rules for valid aliases. Logging has been added as part of the enhancements to make the validation processes easier to understand:\npublic final class AliasPolicy implements HasLogger { public static Validation validate(String alias) { HasLogger.staticLogger().info(\u0026#34;validate - {}\u0026#34;, alias); if (alias == null || alias.isBlank()) return Validation.fail(Reason.NULL_OR_BLANK); if (alias.length() \u0026lt; MIN) return Validation.fail(Reason.TOO_SHORT); if (alias.length() \u0026gt; MAX) return Validation.fail(Reason.TOO_LONG); if (! ALLOWED_PATTERN.matcher(alias).matches()) return Validation.fail(Reason.INVALID_CHARS); return Validation.success(); } } This logging makes faulty aliases immediately visible, making troubleshooting the interaction between the UI and the backend much easier.\nWith these adjustments, the domain model becomes a robust, clearly structured core of the application. The central entity ShortUrlMapping fully reflects the real-world state of a shortlink, while ShortenRequest controls the creation of new entries and provides system-wide constants to DefaultValues. All extensions remain consistent with the original design principle: simple, functional structures that precisely define what a user can create, modify, or retrieve.\nJSON serialisation and deserialization # The reliability of communication between the components of an application depends crucially on the quality of the serialisation layer. This chapter describes how JSON processing has been extended and stabilised to safely transport the new expiresAt field while improving code readability and error tolerance.\nExtending Serialisation in JsonUtils # The JsonUtils class forms the backbone of JSON processing in the project. It provides both generic helper methods and specific serialisation routines for the most critical domain objects. With the introduction of the expiration date, it was necessary to ensure this field was correctly integrated into JSON documents without affecting legacy data formats.\nIn the method for serialising ShortenRequest, the field expiresAt has therefore been added:\nif (dto instanceof ShortenRequest req) { Map\u0026lt;String, Object\u0026gt; m = new LinkedHashMap\u0026lt;\u0026gt;(); m.put(\u0026#34;url\u0026#34;, req.getUrl()); m.put(\u0026#34;alias\u0026#34;, req.getShortURL()); m.put(\u0026#34;expiresAt\u0026#34;, req.getExpiresAt()); return toJson(m); } This change will automatically include the expiration date, if it exists. If it is not set, the field appears as null in the JSON and thus remains syntactically valid. This explicit representation of null values improves readability and allows the server to clearly distinguish between \u0026ldquo;not set\u0026rdquo; and \u0026ldquo;deliberately empty\u0026rdquo;.\nExtending Deserialization # Analogous to serialisation, deserialization has also been extended to read expiresAt from JSON data correctly. In the fromJson method, the customisation is done:\nif (type == ShortenRequest.class) { String url = m.get(\u0026#34;url\u0026#34;); String alias = m.get(\u0026#34;alias\u0026#34;); Instant expiresAt = parseInstantSafe(m.get(\u0026#34;expiresAt\u0026#34;)); return (T) new ShortenRequest(url, alias, expiresAt); } The parseInstantSafe function converts an ISO-8601 string to an instant object and handles invalid or empty values gracefully. This error resistance is significant for clients that may send different or older JSON structures.\nPurge of incoming JSON data # A common problem with APIs is reading in JSON data that contains unintentional line breaks or extra escape characters. To prevent parsing errors, simple preprocessing has been introduced in JsonUtils.parseJson:\ns = s.replaceAll(\u0026quot;\\\\n\u0026quot;, \u0026ldquo;\u0026rdquo;);\nThis step removes all line breaks before the parser runs. This reliably recognises and correctly interprets both manually formatted JSON files and logged messages. This customisation makes the system more robust against inconsistent formatting that is common in real-world environments. (At this point, however, I am aware that it is far from sufficient\u0026hellip;)\nOptimisation of JSON output # As part of these changes, the toJsonListing method has also been revised. Instead of a complicated StringBuilder structure, a simple, readable string concatenation is now used:\nreturn \u0026#34;{\u0026#34; + \u0026#34;\\\u0026#34;mode\\\u0026#34;:\\\u0026#34;\u0026#34; + escape(mode) + \u0026#34;\\\u0026#34;,\u0026#34; + \u0026#34;\\\u0026#34;count\\\u0026#34;:\u0026#34; + count + \u0026#34;,\u0026#34; + \u0026#34;\\\u0026#34;items\\\u0026#34;:\u0026#34; + toJsonArrayOfObjects(items) + \u0026#34;}\u0026#34;; This simplification reduces the susceptibility to errors and makes debugging easier. Especially in systems that do not require frameworks, code readability is a decisive factor for maintainability and error diagnosis.\nConsistency and interoperability # An essential aspect of the revision of serialisation was maintaining interoperability across different clients and API versions. Since all fields are still serialised on a string-by-string basis and the JSON structure is explicit, the data exchange remains fully compatible even with older clients. This means that a client that does not send expiresAt will be accepted by the server, and a server that does not expect the field will ignore it.\nThis loose coupling between transmitter and receiver is a central design goal of the project. It allows incremental expansions without updating all components at once.\nThe revision to JSON processing strengthens the application\u0026rsquo;s robustness and future-proofing. By specifically extending JsonUtils to include the expiresAt field, cleaning incoming data, and simplifying the output, serialisation is now both technically stable and semantically precise. It thus meets the requirements of a modern, evolvable interface that remains clearly comprehensible for both automated processes and human readers.\nSecurity and robustness in the UI flow # As the complexity of the user interface grows, so does the responsibility to ensure that all interactions remain predictable, valid, and stable. This chapter explains how security considerations and robustness were built directly into the UI flow – from input validation to defensive navigation decisions to dealing with external browser APIs.\nInput validations in the generation dialogue # The most important security aspect in the UI is validating user input. As soon as a new short link is created, the application checks whether the entered URL contains a valid scheme. This prevents potential attacks by manipulated or unsupported protocols at an early stage.\nbinder.forField(urlField) .asRequired(\u0026#34;URL must not be empty\u0026#34;) .withValidator(url -\u0026gt; url.startsWith(\u0026#34;http://\u0026#34;) || url.startsWith(\u0026#34;https://\u0026#34;), \u0026#34;Only HTTP(S) URLs allowed\u0026#34;) .bind(ShortenRequest::getUrl, ShortenRequest::setUrl); This simple validation only accepts URLs that can be reached via secure or well-defined transport protocols. This protects both the user and the application from unwanted interactions with unsafe targets. However, it is not yet full input validation.\nAnother part of the validation concerns the expiration date. Here, it is ensured that a selected point in time is always in the future:\nif (exp.isPresent() \u0026amp;\u0026amp; exp.get().isBefore(Instant.now())) { Notification.show(\u0026#34;Expiry must be in the future\u0026#34;); return false; } This mechanism protects against incorrect entries and inconsistent data states, especially when users edit the form multiple times or select times that have already expired.\nDefensive navigation in the detail dialogue # The detail dialogue (DetailsDialog) also follows the principle of secure interaction. If a saved URL is opened via the \u0026ldquo;Open\u0026rdquo; button, this is done exclusively by calling the method UI.getCurrent().getPage().open(), but only if the destination is clearly recognised as an HTTP or HTTPS link. This prevents internal or local resources from being accidentally or intentionally called via the UI.\nopenBtn.addClickListener(_ -\u0026gt; { if (originalUrl.startsWith(\u0026#34;http://\u0026#34;) || originalUrl.startsWith(\u0026#34;https://\u0026#34;)) { fireEvent(new OpenEvent(this, shortCode, originalUrl)); getUI().ifPresent(ui -\u0026gt; ui.getPage().open(originalUrl, \u0026#34;_blank\u0026#34;)); } else { Notification.show(\u0026#34;Invalid URL scheme\u0026#34;); } }); This decision strengthens the separation between internal and external resources. User actions are controlled to prevent unwanted side effects outside the application.\nDealing with the Clipboard API # Another security-relevant topic is how to use the native Clipboard API. For data protection reasons, this is only available in secure browser contexts, i.e. via HTTPS or localhost. The application uses this API to copy shortcodes and URLs to the clipboard conveniently. If access is not allowed in the current context, the call does not result in an error, but is silently discarded – an example of defensive programming behaviour in the UI.\nUI.getCurrent().getPage().executeJs(\u0026ldquo;navigator.clipboard.writeText($0)\u0026rdquo;, SHORTCODE_BASE_URL + m.shortCode());\nThis non-blocking call avoids JavaScript errors and keeps the UI stable even if the browser rejects the action. The application always responds in a controlled manner and remains in a valid state.\nConsistent feedback and fault tolerance # A core element of robustness is the feedback system. Every user command – whether successful or incorrect – triggers visual feedback. The application consistently uses the Vaadin notification component for this purpose. It provides information about validation errors, successful copying operations and system messages without interrupting the workflow. This asynchronous reporting system supports the idea that a user can continue at any time, even if a single operation fails.\nNotification.show(\u0026ldquo;Alias already assigned or error saving\u0026rdquo;, 3000, Notification.Position.MIDDLE);\nThis type of error communication avoids frustration and contributes to perceived stability. The user remains informed, but never blocked.\nThe measures described in this chapter – input validations, defensive navigation, and safe clipboard use – share a common goal: robustness through caution. Every action in the user interface is checked, every external interface is secured, and every user interaction is clearly reported. This allows the UI to achieve a high level of error tolerance without sacrificing ease of use. This balance of security and user-friendliness concludes the technical implementation of the detailed dialogue. It lays the foundation for the upcoming enhancements, in which security and user experience will continue to go hand in hand.\nResult # With today\u0026rsquo;s Advent calendar, the application has reached a decisive degree of maturity. While the first parts mainly laid the structural and functional foundations, today it is all about the depth of detail and the quality of interaction. The focus was on transitioning from a technically functional interface to a well-thought-out, user-centric application.\nThe newly introduced detail dialogue marks a central turning point: it allows viewing individual entries in context without leaving the overview. This concept combines technical clarity with ease of use, creating a modular structure that can easily accommodate future expansions. By using events to decouple the components, the architecture remains clean, traceable and testable.\nThe integration of the expiration dateis also proving to be a milestone in the system\u0026rsquo;s functional development. From input in the UI to the client layer to persistence and JSON processing, the new parameter has been consistently integrated into all layers. The original design was deliberately kept simple – each layer knows only its own responsibility, and none contains logic that anticipates another layer. The result is a consistent, precise data flow that combines technical precision with semantic transparency.\nAnother critical advance is improvingthe user experience (UX). Consistent interaction patterns, precise feedback, and security-conscious behaviour make the application both intuitive and trustworthy. The combination of immediate feedback, non-blocking error handling, and a harmonious visual-functional design shows how technological rigour and user focus can go hand in hand.\nFrom an architectural point of view, this part of the Advent calendar illustrates that even in small projects**, cleanliness, coherence, and expandability** are the key success factors. The clear separation between UI, client, server, and persistence not only enables efficient maintenance but also opens the way for future modules – such as administrative views, bulk operations, or security policies for user groups.\nCheers Sven\n","date":"7 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-detail-dialog-part-2/","section":"Posts","summary":"Client contract from a UI perspective # In this project, the user interface not only serves as a graphical layer on top of the backend, but is also part of the overall contract between the user, the client, and the server. This part focuses on the data flow from the UI’s perspective: how inputs are translated into structured requests, how the client forwards them, and what feedback the user interface processes.\n","title":"Advent Calendar - 2025 - Detail Dialog - Part 2","type":"posts"},{"content":" Classification and objectives from a UI perspective # Today\u0026rsquo;s Advent Calendar Day focuses specifically on the interaction level prepared in the previous parts. While the basic structure of the user interface and the layout were defined at the beginning, and the interactive table view with sorting, filtering and dynamic actions was subsequently established, it is now a matter of making the transition from overview to detailed observation consistent. The user should no longer only see a tabular collection of data points, but should receive a view tailored to the respective object that enables contextual actions.\nClassification and objectives from a UI perspective OverviewView: Interactive enhancements Detail view as a standalone UI component (DetailsDialog) CreateView: Expiration date in the UI Navigation and package structure Interaction patterns and UX coherence Uniform copying behaviour Context menus and multiple interactions Consistency of feedback HTTPS Requirement for Clipboard The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-03\nHere\u0026rsquo;s the screenshot of the version we\u0026rsquo;re implementing now.\nThe central goal of this chapter is to improve the user experience without sacrificing the simplicity of what has been done so far. The interface is extended with a detailed view implemented as a standalone dialogue , thereby deliberately creating a clear framework between the overall list and the individual object. This decision follows the principle of the cognitive separation of information spaces: an overview serves to orient and find relevant entries. At the same time, the detailed view creates an isolated, focused working environment.\nFrom an architectural point of view, this extension is an essential step towards a component-oriented UI in which each functional unit – such as creating, displaying or deleting short links – is represented by its own, clearly defined components. The new dialogue includes all the necessary UI elements, validations, and event flows for displaying and interacting with a single ShortUrlMapping object. This decoupling not only enables better testability and reusability but also reduces the cognitive overhead in the primary grid, which previously had to record all interactions directly.\nThe dialogue serves as an interactive interface between the visual representation and the underlying data model. It reflects the current object, provides copy and navigation actions, and provides visual feedback on the entry\u0026rsquo;s status – for example, via a colour-coded expiration date. This modular structure allows the user to check details, perform actions, or remove erroneous entries without losing the application\u0026rsquo;s context.\nOverviewView: Interactive enhancements # With the third expansion stage of the user interface, the previous list view is not only expanded, but also functionally upgraded. The OverviewView forms the foundation of the daily work with the created short links and is supplemented in this step by several mechanisms that make the behaviour of the application more natural and efficient.\nA central aspect concerns the reactivity of the search fields. In previous versions, search queries were triggered immediately after each input, which was technically correct but inefficient in practice, using the ValueChangeMode.LAZY, in combination with a 400-millisecond delay, the system does not react until the user has completed their input. This delay serves as a natural \u0026ldquo;pause for thought\u0026rdquo; and prevents unnecessary updates to the grid. In addition, the enter-key flow has been activated so that a targeted search confirmation can be performed via the keyboard – a typical usage pattern in the professional environment.\ncodePart.setValueChangeMode(ValueChangeMode.LAZY); codePart.setValueChangeTimeout(400); codePart.addValueChangeListener(e -\u0026gt; refresh()); urlPart.setValueChangeMode(ValueChangeMode.LAZY); urlPart.setValueChangeTimeout(400); urlPart.addValueChangeListener(e -\u0026gt; refresh()); A second improvement concerns the variety of interactions in the grid. Instead of acting exclusively via buttons, the user can now open a data record by double-clicking or pressing Enter. This form of interaction is not only faster, but also meets the expectations you are used to from desktop applications. To further simplify operation, a context menu has been added that automatically offers the appropriate actions when right-clicking: View details, open target URL, copy shortcode, or delete the entry.\ngrid.addItemDoubleClickListener(ev -\u0026gt; openDetailsDialog(ev.getItem())); grid.addItemClickListener(ev -\u0026gt; { if (ev.getClickCount() == 2) openDetailsDialog(ev.getItem()); }); GridContextMenu\u0026lt;ShortUrlMapping\u0026gt; menu = new GridContextMenu\u0026lt;\u0026gt;(grid); menu.addItem(\u0026#34;Show details\u0026#34;, e -\u0026gt; e.getItem().ifPresent(this::openDetailsDialog)); menu.addItem(\u0026#34;Open URL\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().open(m.originalUrl(), \u0026#34;_blank\u0026#34;))); menu.addItem(\u0026#34;Copy shortcode\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, m.shortCode()))); menu.addItem(\u0026#34;Delete...\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; confirmDelete(m.shortCode()))); In addition to functionality, the grid column layout has been revised. The shortcode is now displayed in a monospace font , which makes it easier to quickly grasp and compare visually. In addition, a small copy symbol allows copying the entire short link to the clipboard with a mouse click. JavaScript is used to call up the browser\u0026rsquo;s native clipboard service.\ngrid.addComponentColumn(m -\u0026gt; { var code = new Span(m.shortCode()); code.getStyle().set(\u0026#34;font-family\u0026#34;, \u0026#34;ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace\u0026#34;); var copy = new Button(new Icon(VaadinIcon.COPY)); copy.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE); copy.getElement().setProperty(\u0026#34;title\u0026#34;, \u0026#34;Copy ShortUrl\u0026#34;); copy.addClickListener(_ -\u0026gt; { UI.getCurrent().getPage().executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, SHORTCODE_BASE_URL + m.shortCode()); Notification.show(\u0026#34;Shortcode copied\u0026#34;); }); var wrap = new HorizontalLayout(code, copy); wrap.setSpacing(true); wrap.setPadding(false); return wrap; }).setHeader(\u0026#34;Shortcode\u0026#34;).setAutoWidth(true).setFrozen(true).setResizable(true).setFlexGrow(0); In the column with the original URL, the layout has been changed to an elliptical display : Long URLs are truncated, but remain fully visible via the tooltip. This detail improves readability without losing information.\ngrid.addComponentColumn(m -\u0026gt; { var a = new Anchor(m.originalUrl(), m.originalUrl()); a.setTarget(\u0026#34;_blank\u0026#34;); a.getStyle() .set(\u0026#34;white-space\u0026#34;, \u0026#34;nowrap\u0026#34;) .set(\u0026#34;overflow\u0026#34;, \u0026#34;hidden\u0026#34;) .set(\u0026#34;text-overflow\u0026#34;, \u0026#34;ellipsis\u0026#34;) .set(\u0026#34;display\u0026#34;, \u0026#34;inline-block\u0026#34;) .set(\u0026#34;max-width\u0026#34;, \u0026#34;100%\u0026#34;); a.getElement().setProperty(\u0026#34;title\u0026#34;, m.originalUrl()); return a; }).setHeader(\u0026#34;URL\u0026#34;).setFlexGrow(1).setResizable(true); Particularly noteworthy is the new Expires column , which colour-coded status indicators have visually supplemented. These are based on Lumo badges and show the remaining time until a link expires: green for active, yellow for expiring soon, and red for expired.\ngrid.addComponentColumn(m -\u0026gt; { var pill = new Span(m.expiresAt() .map(ts -\u0026gt; { var days = Duration.between(Instant.now(), ts).toDays(); if (days \u0026lt; 0) return \u0026#34;Expired\u0026#34;; if (days == 0) return \u0026#34;Today\u0026#34;; return \u0026#34;in \u0026#34; + days + \u0026#34; days\u0026#34;; }).orElse(\u0026#34;No expiry\u0026#34;)); pill.getElement().getThemeList().add(\u0026#34;badge pill small\u0026#34;); m.expiresAt().ifPresent(ts -\u0026gt; { long d = Duration.between(Instant.now(), ts).toDays(); if (d \u0026lt; 0) pill.getElement().getThemeList().add(\u0026#34;error\u0026#34;); else if (d \u0026lt;= 3) pill.getElement().getThemeList().add(\u0026#34;warning\u0026#34;); else pill.getElement().getThemeList().add(\u0026#34;success\u0026#34;); }); return pill; }).setHeader(\u0026#34;Expires\u0026#34;).setAutoWidth(true).setResizable(true).setFlexGrow(0); Finally, the display\u0026rsquo;s ergonomics have been improved. The grid now works with reduced vertical spacing (compact mode) while retaining its readability thanks to alternating line colours. The line height has been adjusted so that as many entries as possible remain visible even on smaller monitors without making the UI look overloaded.\nWith these changes, the OverviewView is transformed from a purely management view into a central control tool that provides both a quick overview and deep interaction. This lays the foundation for integrating the detail dialogue – it complements this view with an object-related perspective, while the OverviewView continues to serve as the starting point for navigation.\nDetail view as a standalone UI component (DetailsDialog) # Now that the overview list has been expanded with interactive functions, the next logical step is to introduce a standalone UI component focused on the display and management of individual datasets. The goal is to create a modular and reusable view that operates independently of the main view but communicates with it via clearly defined events.\nThe dialogue is based on Vaadin\u0026rsquo;s Dialogue class and opens for each selected ShortUrlMapping object. In doing so, it reads out all relevant properties of the passed object – shortcode, target URL, creation time and, optionally, the expiration date. These values are presented in a clearly structured format, supplemented by action buttons to open, copy and delete the entry.\nThe following excerpt shows the basic structure of the dialogue:\npublic class DetailsDialog extends Dialog implements HasLogger { public static final ZoneId ZONE = ZoneId.systemDefault(); private static final DateTimeFormatter DATE_TIME_FMT = DateTimeFormatter.ofPattern(\u0026#34;dd.MM.yyyy HH:mm\u0026#34;).withZone(ZONE); private final String shortCode; private final String originalUrl; private final Instant createdAt; private final Optional\u0026lt;Instant\u0026gt; expiresAt; private final TextField tfShort = new TextField(\u0026#34;Shortcode\u0026#34;); private final TextField tfUrl = new TextField(\u0026#34;Original URL\u0026#34;); private final TextField tfCreated = new TextField(\u0026#34;Created on\u0026#34;); private final TextField tfExpires = new TextField(\u0026#34;Expires\u0026#34;); private final Span statusPill = new Span(); private final Button openBtn = new Button(\u0026#34;Open\u0026#34;, new Icon(VaadinIcon.EXTERNAL_LINK)); private final Button copyShortBtn = new Button(\u0026#34;Copy ShortURL\u0026#34;, new Icon(VaadinIcon.COPY)); private final Button copyUrlBtn = new Button(\u0026#34;Copy URL\u0026#34;, new Icon(VaadinIcon.COPY)); private final Button deleteBtn = new Button(\u0026#34;Delete...\u0026#34;, new Icon(VaadinIcon.TRASH)); private final Button closeBtn = new Button(\u0026#34;Close\u0026#34;); public DetailsDialog(ShortUrlMapping mapping) { Objects.requireNonNull(mapping, \u0026#34;mapping\u0026#34;); this.shortCode = mapping.shortCode(); this.originalUrl = mapping.originalUrl(); this.createdAt = mapping.createdAt(); this.expiresAt = mapping.expiresAt(); setHeaderTitle(\u0026#34;Details: \u0026#34; + shortCode); setModal(true); setDraggable(true); setResizable(true); setWidth(\u0026#34;720px\u0026#34;); openBtn.addThemeVariants(ButtonVariant.LUMO_PRIMARY); deleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR, ButtonVariant.LUMO_TERTIARY); var headerActions = new HorizontalLayout(openBtn, copyShortBtn, copyUrlBtn, deleteBtn); getHeader().add(headerActions); configureFields(); var form = new FormLayout(); form.add(tfShort, tfUrl, tfCreated, tfExpires, statusPill); form.setColspan(tfUrl, 2); add(form); closeBtn.addClickListener(e -\u0026gt; close()); getFooter().add(closeBtn); wireActions(); } It is already clear here that the dialogue has a high degree of independence. It initialises its contents directly from the passed data object and uses a FormLayout to structure the fields. The input fields are read-only by default because the dialogue is used primarily for display.\nThe visual feedback on the expiration status is provided by a so-called status pill , which indicates in colour and text whether a short link is still active, about to expire or has already expired. This is done via a small helper method that provides a status description including a colour scheme:\nprivate status computeStatusText() { return expiresAt.map(ts -\u0026gt; { long d = Duration.between(Instant.now(), ts).toDays(); if (d \u0026lt; 0) return new Status(\u0026#34;Expired\u0026#34;, \u0026#34;error\u0026#34;); if (d == 0) return new Status(\u0026#34;Expires today\u0026#34;, \u0026#34;warning\u0026#34;); if (d \u0026lt;= 3) return new Status(\u0026#34;Expires in \u0026#34; + d + \u0026#34; days\u0026#34;, \u0026#34;warning\u0026#34;); return new Status(\u0026#34;Valid(\u0026#34; + d + \u0026#34; days left)\u0026#34;, \u0026#34;success\u0026#34;); }).orElse(new Status(\u0026#34;No expiry\u0026#34;, \u0026#34;contrast\u0026#34;)); } This status calculation complements the visual feedback from the summary table and provides a consistent representation across the entire application.\nThe real added value of the DetailsDialog lies in its event-orientation. Instead of the calling view (OverviewView) controlling all actions itself, the actions are defined as Vaadin events. This allows the dialogue to send signals such as OpenEvent , CopyShortcodeEvent , CopyUrlEvent or DeleteEvent to its environment without knowing what they mean:\npublic static class DeleteEvent extends ComponentEvent\u0026lt;DetailsDialog\u0026gt; { public final String shortCode; public DeleteEvent(DetailsDialog src, String sc) { super(src); this.shortCode = sc; } } In the OverviewView , these events are received and processed:\nvar dlg = new DetailsDialog(item); dlg.addDeleteListener(ev -\u0026gt; confirmDelete(ev.shortCode)); dlg.addOpenListener(ev -\u0026gt; logger().info(\u0026#34;Open URL {}\u0026#34;, ev.originalUrl)); dlg.addCopyShortListener(ev -\u0026gt; logger().info(\u0026#34;Copied shortcode {}\u0026#34;, ev.shortCode)); dlg.addCopyUrlListener(ev -\u0026gt; logger().info(\u0026#34;Copied URL {}\u0026#34;, ev.url)); dlg.open(); This creates an apparent decoupling between the presentation and application logic. Dialogue handles representation and interaction; the surrounding view determines how to respond to events.\nIn summary, the DetailsDialog establishes an architectural principle that goes beyond the specific use case. The combination of a modular UI component, declarative events and a clearly defined data model not only makes the application more flexible, but also more maintainable in the long term. The user benefits from a coherent, clearly structured, detailed view, which makes working with individual entries much more convenient and transparent.\nCreateView: Expiration date in the UI # With this step, the creation of new short links receives an important semantic addition: the optional expiration date. The aim is to precisely define the intended service life at the time of creation and to transport this information end-to-end – from the UI to the client to the server and into the persistence.\nFrom a UI point of view, the existing CreateView is extended with DatePicker and TimePicker, flanked by a checkbox labelled \u0026ldquo;No expiry\u0026rdquo;. This combination allows both the explicit determination of an end date and the conscious decision to set an unlimited validity. A small but crucial UX rule: the time will remain disabled until a date is selected, and both fields will be disabled if \u0026ldquo;No expiry\u0026rdquo; is enabled.\nFields and Basic Configuration\nprivate final TextField urlField = new TextField(\u0026#34;Target URL\u0026#34;); private final TextField aliasField = new TextField(\u0026#34;Alias (optional)\u0026#34;); private final Button shortenButton = new Button(\u0026#34;Shorten\u0026#34;); private final DatePicker expiresDate = new DatePicker(\u0026#34;Expires (date)\u0026#34;); private final TimePicker expiresTime = new TimePicker(\u0026#34;Expires (time)\u0026#34;); private final Checkbox noExpiry = new Checkbox(\u0026#34;No expiry\u0026#34;); private final FormLayout form = new FormLayout(); public CreateView() { setSpacing(true); setPadding(true); urlField.setWidthFull(); aliasField.setWidth(\u0026#34;300px\u0026#34;); shortenButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY); form.add(urlField, aliasField); configureExpiryFields(); form.setResponsiveSteps( new FormLayout.ResponsiveStep(\u0026#34;0\u0026#34;, 1), new FormLayout.ResponsiveStep(\u0026#34;600px\u0026#34;, 2) ); form.setColspan(urlField, 2); var actions = new HorizontalLayout(shortenButton); actions.setAlignItems(Alignment.END); Binder for validations Binder\u0026lt;ShortenRequest\u0026gt; binder = new Binder\u0026lt;\u0026gt;(ShortenRequest.class); ShortenRequest request = new ShortenRequest(); binder.forField(urlField) .asRequired(\u0026#34;URL must not be empty\u0026#34;) .withValidator(url -\u0026gt; url.startsWith(\u0026#34;http://\u0026#34;) || url.startsWith(\u0026#34;https://\u0026#34;), \u0026#34;Only HTTP(S) URLs allowed\u0026#34;) .bind(ShortenRequest::getUrl, ShortenRequest::setUrl); binder.forField(aliasField) .withValidator(a -\u0026gt; a == null || a.isBlank() || a.length() \u0026lt;= AliasPolicy.MAX, \u0026#34;Alias is too long (max \u0026#34; + AliasPolicy.MAX + \u0026#34;)\u0026#34;) .withValidator(a -\u0026gt; a == null || a.isBlank() || a.matches(REGEX_ALLOWED), \u0026#34;Only [A-Za-z0-9_-] allowed\u0026#34;) .bind(ShortenRequest::getShortURL, ShortenRequest::setShortURL); shortenButton.addClickListener(_ -\u0026gt; { var validated = binder.validate(); if (validated.hasErrors()) return; if (!validateExpiryInFuture()) return; if (binder.writeBeanIfValid(request)) { computeExpiresAt().ifPresent(request::setExpiresAt); var code = createShortCode(request, computeExpiresAt()); code.ifPresentOrElse(c -\u0026gt; { Notification.show(\u0026#34;Short link created: \u0026#34; + c); clearForm(binder); }, () -\u0026gt; Notification.show(\u0026#34;Alias already assigned or error saving\u0026#34;, 3000, Notification.Position.MIDDLE)); } }); add(new H2(\u0026#34;Create new short link\u0026#34;), form, actions); } Encapsulating process logic in small auxiliary methods improves readability and testability. The calculation uses the local time zone and provides an instant that can be processed unchanged on the server side.\nprivate static final ZoneId ZONE = ZoneId.systemDefault(); private void configureExpiryFields() { expiresDate.setClearButtonVisible(true); expiresDate.setPlaceholder(\u0026#34;dd.MM.yyyy\u0026#34;); expiresTime.setStep(Duration.ofMinutes(1)); expiresTime.setPlaceholder(\u0026#34;HH:mm\u0026#34;); Activate time only when a date is set expiresTime.setEnabled(false); expiresDate.addValueChangeListener(ev -\u0026gt; { boolean hasDate = ev.getValue() != null; expiresTime.setEnabled(hasDate \u0026amp;\u0026amp; !noExpiry.getValue()); }); noExpiry.addValueChangeListener(ev -\u0026gt; { boolean disabled = ev.getValue(); expiresDate.setEnabled(!disabled); expiresTime.setEnabled(!disabled \u0026amp;\u0026amp; expiresDate.getValue() != null); }); form.add(noExpiry, expiresDate, expiresTime); } private Optional\u0026lt;Instant\u0026gt; computeExpiresAt() { if (Boolean.TRUE.equals(noExpiry.getValue())) return Optional.empty(); LocalDate d = expiresDate.getValue(); LocalTime t = expiresTime.getValue(); if (d == null || t == null) return Optional.empty(); return Optional.of(ZonedDateTime.of(d, t, ZONE).toInstant()); } private boolean validateExpiryInFuture() { var exp = computeExpiresAt(); if (exp.isPresent() \u0026amp;\u0026amp; exp.get().isBefore(Instant.now())) { Notification.show(\u0026#34;Expiry must be in the future\u0026#34;); return false; } return true; } private void clearForm(Binder\u0026lt;ShortenRequest\u0026gt; binder) { urlField.clear(); aliasField.clear(); noExpiry.clear(); expiresDate.clear(); expiresTime.clear(); binder.setBean(new ShortenRequest()); urlField.setInvalid(false); aliasField.setInvalid(false); } It is important to pass it on transparently to the client. Instead of storing the time in the UI, it is transmitted to the server in the request. To do this, the CreateView calls the client with the extended signature. The client validates only if an alias is set and passes the data to the server unchanged.\nprivate Optional\u0026lt;String\u0026gt; createShortCode(ShortenRequest req, Optional\u0026lt;Instant\u0026gt; expiresAt) { logger().info(\u0026#34;createShortCode with ShortenRequest \u0026#39;{}\u0026#39;\u0026#34;, req); try { var customMapping = urlShortenerClient.createCustomMapping(req.getShortURL(), req.getUrl(), expiresAt.orElse(null)); return Optional.ofNullable(customMapping.shortCode()); } catch (IllegalArgumentException | IOException e) { logger().error(\u0026#34;Error saving\u0026#34;, e); return Optional.empty(); } } On the client side , the payload is serialised as a ShortenRequest and sent to the /shorten endpoint. The response is not limited to the shortcode; it is also parsed as a full ShortUrlMapping. As a result, the UI immediately knows the server-side confirmed state – including the expiresAt.\npublic ShortUrlMapping createCustomMapping(String alias, String url, Instant expiredAt) throws IOException { logger().info(\u0026#34;Create custom mapping alias=\u0026#39;{}\u0026#39; url=\u0026#39;{}\u0026#39; expiredAt=\u0026#39;{}\u0026#39;\u0026#34;, alias, url, expiredAt); if (alias != null \u0026amp;\u0026amp; !alias.isBlank()) { var validate = AliasPolicy.validate(alias); if (!validate.valid()) { var reason = validate.reason(); throw new IllegalArgumentException(reason.defaultMessage); } } URL shortenUrl = serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL(); HttpURLConnection connection = (HttpURLConnection) shortenUrl.openConnection(); connection.setRequestMethod(\u0026#34;POST\u0026#34;); connection.setDoOutput(true); connection.setRequestProperty(CONTENT_TYPE, JSON_CONTENT_TYPE); var shortenRequest = new ShortenRequest(url, alias, expiredAt); String body = shortenRequest.toJson(); try (OutputStream os = connection.getOutputStream()) { os.write(body.getBytes(UTF_8)); } int status = connection.getResponseCode(); if (status == 200 || status == 201) { try (InputStream is = connection.getInputStream()) { String jsonResponse = new String(is.readAllBytes(), UTF_8); ShortUrlMapping shortUrlMapping = fromJson(jsonResponse, ShortUrlMapping.class); return shortUrlMapping; } } if (status == 409) { throw new IllegalArgumentException(\u0026#34;Alias already in use\u0026#34;); } throw new IOException(\u0026#34;Unexpected status: \u0026#34; + status); } On the server side , the ShortenHandler receives the extended request, validates the required fields, and then assigns the store to create it. The response contains the complete mapping object that the client and UI can use immediately.\nfinal String body = readBody(ex.getRequestBody()); ShortenRequest req = fromJson(body, ShortenRequest.class); if (isNullOrBlank(req.getUrl())) { writeJson(ex, BAD_REQUEST, \u0026#34;Missing \u0026#39;url\u0026#39;\u0026#34;); return; } final Result\u0026lt;ShortUrlMapping\u0026gt; urlMappingResult = store.createMapping(req.getShortURL(), req.getUrl(), req.getExpiresAt()); urlMappingResult .ifPresentOrElse(success -\u0026gt; logger().info(\u0026#34;mapping created success {}\u0026#34;, success.toString()), failed -\u0026gt; logger().info(\u0026#34;mapping created failed - {}\u0026#34;, failed)); urlMappingResult .ifSuccess(mapping -\u0026gt; { final Headers h = ex.getResponseHeaders(); h.add(\u0026#34;Location\u0026#34;, \u0026#34;/r/\u0026#34; + mapping.shortCode()); writeJson(ex, fromCode(201), toJson(mapping)); }) .ifFailure(errorJson -\u0026gt; { try { var parsed = JsonUtils.parseJson(errorJson); var errorCode = Integer.parseInt(parsed.get(\u0026#34;code\u0026#34;)); var message = parsed.get(\u0026#34;message\u0026#34;); writeJson(ex, fromCode(errorCode), message); } catch (Exception e) { writeJson(ex, CONFLICT, errorJson); } }); In summary**, a consistent, end-to-end process** is created: The user optionally defines an expiration date when making them, the UI validates basic rules, the client transfers the semantics losslessly, and the server persists them reliably. The OverviewView and the DetailsDialog can then display and interpret this information immediately. This adds a central property to the domain without complicating the existing operating flow.\nNavigation and package structure # The previous user interface has become much more complex in terms of content due to the detailed dialogue and the extended form functions. To reflect this development, the package structure in the UI module was clearly reorganised. The goal was not only a logical grouping according to responsibilities, but also a long-term basis for extended navigation concepts and modular extensions.\nPreviously, the OverviewView was still in the general package com.svenruppert.urlshortener.ui.vaadin.views. With the introduction of detailed dialogue and the growing importance of this area in terms of content, it was included in a separate subpackage, \u0026ldquo;views.overview \u0026ldquo;, and moved there. This decision not only creates room for additional components (e.g. context menus, helper dialogues or filters), but also follows the principle of functional coherence : All classes that together form the overview function are now centrally bundled.\nIn the code, this step is reflected in the customisation of the import within the MainLayout :\nold:\nimport com.svenruppert.urlshortener.ui.vaadin.views.OverviewView;\nnew:\nimport com.svenruppert.urlshortener.ui.vaadin.views.overview.OverviewView;\nThis small but significant step marks the transition from a purely page-based structure to a component-oriented structure. The MainLayout continues to serve as the central navigation element of the application; the individual views are now more decoupled and can be further developed independently. This creates a clear separation between layout logic (central navigation, menus, visual frameworks) and functional logic (display, interaction, data flow).\nThe route relationships are deliberately kept simple. The OverviewView is still registered under the path /overview and uses the MainLayout as its parent layout element:\n@PageTitle Overview @Route(value = OverviewView.PATH, layout = MainLayout.class) public class OverviewView extends VerticalLayout implements HasLogger { public static final String PATH = \u0026#34;overview\u0026#34;; // ... } This configuration keeps navigation consistent with the other parts of the application, such as CreateView, AboutView, or YoutubeView. New views can be easily added without affecting the central navigation mechanism. This ensures maintainability and scalability – two central requirements for an application that grows gradually within the Advent calendar framework.\nThe MainLayout itself is largely unchanged, but has been updated during the package migration to reflect new imports and menu entries. The referencing of the OverviewView is particularly important, as it represents the entry point of user interaction:\nSideNavItem overview = new SideNavItem(\u0026#34;Overview\u0026#34;, OverviewView.class, VaadinIcon.LIST.create()); SideNavItem create = new SideNavItem(\u0026#34;Create\u0026#34;, CreateView.class, VaadinIcon.PLUS.create()); SideNavItem about = new SideNavItem(\u0026#34;About\u0026#34;, AboutView.class, VaadinIcon.INFO_CIRCLE.create()); SideNavItem youtube = new SideNavItem(\u0026#34;Youtube\u0026#34;, YoutubeView.class, VaadinIcon.YOUTUBE.create()); SideNav nav = new SideNav(overview, create, about, youtube); addToDrawer(nav); This clean separation between routing, structure and presentation lays the foundation for the further development of the user interface. Future features – such as detailed filters, user preferences or administrative functions – can be easily integrated into their own subpackages and namespaces without affecting core navigation. This is the step towards a sustainable UI architecture that specifically supports both growing complexity and extensibility.\nInteraction patterns and UX coherence # As the application\u0026rsquo;s feature density increases, the importance of a consistent user experience grows. While the first days of the Advent calendar laid the technical foundation, so far, the focus has been on functionality. In this chapter, the focus shifts to the coherence of interaction patterns , i.e. how users interact with the application in a consistent, predictable rhythm.\nCentral to this is the goal of establishing uniform behaviour for recurring actions. Whether in the overview, in the detail dialogue or in forms – copying, opening or deleting should always feel the same. The application thus conveys reliability, which is particularly crucial for technical users of web-based tools.\nUniform copying behaviour # A good example is copying URLs and short links. The exact mechanisms are used in both the grid and the detail dialogue: a button with the VaadinIcon.COPY symbol: an asynchronous clipboard action via JavaScript and a discreet confirmation message. This avoids users having to learn different forms of interaction in various views.\ncopy.addClickListener(_ -\u0026gt; { UI.getCurrent().getPage().executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, SHORTCODE_BASE_URL + m.shortCode()); Notification.show(\u0026#34;Shortcode copied\u0026#34;); }); Such a detail may seem inconspicuous, but it has a significant impact on the perceived professionalism of the application. The feedback system via notifications plays a central role here: The user receives an immediate, unobtrusive signal about the success of his action – a kind of visual receipt that creates trust.\nContext menus and multiple interactions # Another element of UX coherence is the context menu in the summary table. It allows access to the same actions, which can also be accessed via buttons or double-clicks. This redundant but deliberate multiple interaction follows the principle of user freedom : Users can choose whether to act via direct icons, the keyboard or the context menu.\nGridContextMenu\u0026lt;ShortUrlMapping\u0026gt; menu = new GridContextMenu\u0026lt;\u0026gt;(grid); menu.addItem(\u0026#34;Show details\u0026#34;, e -\u0026gt; e.getItem().ifPresent(this::openDetailsDialog)); menu.addItem(\u0026#34;Open URL\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().open(m.originalUrl(), \u0026#34;_blank\u0026#34;))); menu.addItem(\u0026#34;Copy shortcode\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; UI.getCurrent().getPage().executeJs(\u0026#34;navigator.clipboard.writeText($0)\u0026#34;, m.shortCode()))); menu.addItem(\u0026#34;Delete...\u0026#34;, e -\u0026gt; e.getItem().ifPresent(m -\u0026gt; confirmDelete(m.shortCode()))); The decision to use context menus is not only aesthetic but also ergonomic: it reduces the grid’s optical density without sacrificing functionality. Actions only appear when they are needed – an approach that is essential in complex management interfaces.\nConsistency of feedback # A uniform pattern is also followed for error messages and validations. The system does not abruptly abort user actions; instead, it clearly communicates why an input was not accepted. Example: The expiration date cannot be in the past. This rule is conveyed both visually and by message.\nif (exp.isPresent() \u0026amp;\u0026amp; exp.get().isBefore(Instant.now())) { Notification.show(\u0026#34;Expiry must be in the future\u0026#34;); return false; } Accurate feedback prevents the user from perceiving the system as unpredictable. Every validation, every hint, and every success message follows the same communicative style—short, clear, and polite.\nHTTPS Requirement for Clipboard # A technical but essential detail is that the Clipboard API only works in modern browsers within secure contexts (HTTPS or localhost). Therefore, the copy feature is deliberately designed to fail elegantly when clipboard access is unavailable. The application does not crash, but responds silently – an aspect that underlines the robustness and professionalism of the user experience.\nThe mechanisms described in this chapter – consistent feedback, redundant operating options and secure fallbacks – together contribute to a coherent user experience. They form the basis of trust and predictability, two characteristics that are essential as software becomes more complex. Overall, the result is an interface that can be operated intuitively, even as its technical depth in the background continues to increase.\nCheers Sven\n","date":"6 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-detail-dialog-part-1/","section":"Posts","summary":"Classification and objectives from a UI perspective # Today’s Advent Calendar Day focuses specifically on the interaction level prepared in the previous parts. While the basic structure of the user interface and the layout were defined at the beginning, and the interactive table view with sorting, filtering and dynamic actions was subsequently established, it is now a matter of making the transition from overview to detailed observation consistent. The user should no longer only see a tabular collection of data points, but should receive a view tailored to the respective object that enables contextual actions.\n","title":"Advent Calendar - 2025 - Detail Dialog - Part 1","type":"posts"},{"content":"Today, we will finally integrate the StoreIndicator into the UI.\nVaadin integration: live status of the store Implementation of the StoreIndicator Refactoring inside – The MappingCreator as a central logic. EclipseStore – The Persistent Foundation Additional improvements in the core Before \u0026amp; After – Impact on the developer experience The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-02\nHere\u0026rsquo;s the screenshot of the version we\u0026rsquo;re implementing now.\nVaadin integration: live status of the store # Now that communication between the client and server has been established via the AdminClient, the focus is on integrating with the Vaadin interface. The goal is to make the current system state visible to the user without requiring manual updates. The mechanism is based on cyclic querying, event control, and a reactive UI concept, all fully implemented in Vaadin.\nA central component of this integration is the StoreIndicator component. It serves as a visual interface between the user and the system state. Its design combines UI representation, data retrieval, and state management. The structure is implemented within the Vaadin component model, with the indicator implemented as an independent class, \u0026ldquo;StoreIndicator\u0026rdquo;. This class extends HorizontalLayout and adds several sub-elements to the layout—an icon, a text label, and optional status information. Their behaviour is closely linked to Vaadin\u0026rsquo;s lifecycle control.\nThe central entry point is the onAttach method, which is executed whenever the component is added to the UI tree. At this moment, the polling mechanism starts, calling the AdminClient every 10 seconds to get the current status from the server. Communication remains fully controlled on the server side, while the UI is automatically synchronised.\n@Override protected void onAttach(AttachEvent attachEvent) { refreshOnce(); UI ui = attachEvent.getUI(); ui.setPollInterval(10000); ui.addPollListener(e -\u0026gt; refreshOnce()); } Analogous to the onAttach method, onDetach is also an essential part of the StoreIndicator\u0026rsquo;s lifecycle. This method is executed when the component is removed from the UI tree, such as when switching between views or closing the session. Here, the polling mechanism is cleanly terminated, and resources such as event subscriptions or listeners are released. This prevents background processes from continuing to send data to a defunct UI component. The code illustrates this process:\n@Override protected void onDetach(DetachEvent detachEvent) { super.onDetach(detachEvent); if (subscription != null) { try { subscription.close(); } catch (Exception ignored) { } subscription = null; } } This implementation ensures that no polling tasks or event listeners remain active after the component is removed. This is especially crucial in server-side frameworks like Vaadin to avoid leaks and unnecessary thread activity. The StoreIndicator thus acts like a well-behaved UI citizen, completing its tasks properly when leaving the scene.\nEach polling cycle triggers the refreshOnce method, which uses the AdminClient to determine the current memory state. This checks whether the server returns the mode \u0026ldquo;EclipseStore\u0026rdquo; or \u0026ldquo;InMemory\u0026rdquo;. The colour scheme is adjusted accordingly, and the symbol is highlighted. The design is based on Vaadin\u0026rsquo;s Lumo theme system, which enables clean integration with the application\u0026rsquo;s corporate design via CSS variables.\nAnother central element is the internal event system, implemented via the StoreEvents and StoreConnectionChanged classes. It allows for loose coupling between components, so that any change to the memory state can be published globally and consumed by other UI elements. The StoreIndicator uses this system to propagate state changes, while the OverviewView, for example, acts as a subscriber and automatically updates itself when the memory state changes.\n**StoreEvents.publish(new StoreConnectionChanged(newMode, info.mappings()));\nThis architecture avoids direct dependencies between UI components. Instead of explicit references or state propagation via view constructors, an event model is used that reduces complexity while increasing the application\u0026rsquo;s responsiveness. The result is a highly modular, reactive UI structure that reflects data changes on the backend.\nThe way in which Vaadin is used here as a reactive framework deserves special attention. While classic web frameworks propagate state changes via HTTP requests and complete page reloads, Vaadin uses server-side push to deliver targeted UI updates. In combination with the polling model of the StoreIndicator, a hybrid solution is created: On the one hand, the architecture remains comprehensible and straightforward, and on the other hand, it gives the impression of a live application that reacts to changes in real time.\nThis integration of event model, UI lifecycle, and data transfer is an example of modern component-oriented design in the Java ecosystem. It shows that reactive behaviour patterns can also be implemented without external reactive frameworks such as Vert.x or Reactor.\nImplementation of the StoreIndicator # The indicator\u0026rsquo;s graphical structure is already fully defined in the constructor. It extends HorizontalLayout and uses Vaadin components such as Icon and Span to create a compact, semantically transparent status display. The structure follows the typical Vaadin composition logic: The graphic elements are created, styled and then inserted into the layout via the add method. Particular attention is paid to legibility and visual consistency:\npublic StoreIndicator(AdminClient adminClient) { this.adminClient = adminClient; setAlignItems(FlexComponent.Alignment.CENTER); setSpacing(true); setPadding(false); dbIcon.setSize(\u0026#34;16px\u0026#34;); dbIcon.getStyle().set(\u0026#34;color\u0026#34;, \u0026#34;var(--lumo-secondary-text-color)\u0026#34;); badge.getStyle() .set(\u0026#34;font-size\u0026#34;, \u0026#34;12px\u0026#34;) .set(\u0026#34;font-weight\u0026#34;, \u0026#34;600\u0026#34;) .set(\u0026#34;padding\u0026#34;, \u0026#34;0.2rem 0.5rem\u0026#34;) .set(\u0026#34;border-radius\u0026#34;, \u0026#34;0.4rem\u0026#34;) .set(\u0026#34;background-color\u0026#34;, \u0026#34;var(--lumo-contrast-10pct)\u0026#34;) .set(\u0026#34;color\u0026#34;, \u0026#34;var(--lumo-body-text-color)\u0026#34;); details.getStyle() .set(\u0026#34;font-size\u0026#34;, \u0026#34;12px\u0026#34;) .set(\u0026#34;opacity\u0026#34;, \u0026#34;0.8\u0026#34;); add(dbIcon, badge, details); } The actual logic of the status update is contained in the refreshOnce method. It calls the AdminClient to determine the current server state. This returns a StoreInfo object containing the mode and the number of saved mappings. The mode dynamically adjusts the indicator\u0026rsquo;s colour scheme– green for persistent, blue for volatile, and red for erroneous states. This colour coding is based on Vaadin\u0026rsquo;s Lumo theming and provides immediate visual feedback.\npublic void refreshOnce() { getUI().ifPresent(ui -\u0026gt; ui.access(() -\u0026gt; { try { StoreInfo info = adminClient.getStoreInfo(); boolean persistent = \u0026#34;EclipseStore\u0026#34;.equalsIgnoreCase(info.mode()); badge.setText(persistent ? \u0026#34;EclipseStore\u0026#34;: \u0026#34;InMemory\u0026#34;); if (persistent) { badge.getStyle() .set(\u0026#34;background-color\u0026#34;, \u0026#34;var(--lumo-success-color-10pct)\u0026#34;) .set(\u0026#34;color\u0026#34;, \u0026#34;var(--lumo-success-text-color)\u0026#34;); dbIcon.getStyle().set(\u0026#34;color\u0026#34;, \u0026#34;var(--lumo-success-color)\u0026#34;); } else { badge.getStyle() .set(\u0026#34;background-color\u0026#34;, \u0026#34;var(--lumo-primary-color-10pct)\u0026#34;) .set(\u0026#34;color\u0026#34;, \u0026#34;var(--lumo-primary-text-color)\u0026#34;); dbIcon.getStyle().set(\u0026#34;color\u0026#34;, \u0026#34;var(--lumo-primary-color)\u0026#34;); } details.setText(\u0026#34;· \u0026#34; + info.mappings() + \u0026#34; items\u0026#34;); getElement().setAttribute(\u0026#34;title\u0026#34;, persistent ? \u0026#34;Persistent via EclipseStore\u0026#34; : \u0026#34;Volatile (InMemory)\u0026#34;); var newMode = persistent ? StoreMode.ECLIPSE_STORE : StoreMode.IN_MEMORY; if (newMode != lastMode) { lastMode = newMode; StoreEvents.publish(new StoreConnectionChanged(newMode, info.mappings())); } } catch (Exception e) { badge.setText(\u0026#34;Unavailable\u0026#34;); badge.getStyle() .set(\u0026#34;background-color\u0026#34;, \u0026#34;var(--lumo-error-color-10pct)\u0026#34;) .set(\u0026#34;color\u0026#34;, \u0026#34;var(--lumo-error-text-color)\u0026#34;); dbIcon.getStyle().set(\u0026#34;color\u0026#34;, \u0026#34;var(--lumo-error-color)\u0026#34;); details.setText(\u0026#34;\u0026#34;); getElement().setAttribute(\u0026#34;title\u0026#34;, \u0026#34;StoreInfo endpoint unavailable\u0026#34;); if (lastMode != StoreMode.UNAVAILABLE) { lastMode = StoreMode.UNAVAILABLE; StoreEvents.publish(new StoreConnectionChanged(StoreMode.UNAVAILABLE, 0)); } } })); } This method is an example of reactive UI design without external frameworks. The combination of ui.access() and PollListener allows periodic updates without blocking the main UI thread model. Error cases are elegantly intercepted via the catch block and propagated via the event bus (StoreEvents.publish), so that other components can also react to failures.\nRefactoring inside – The MappingCreator as a central logic. # With the introduction of EclipseStore and the parallel continued existence of the InMemory store, the need arose to standardise the generation logic for short URLs. Earlier versions of the application included this logic multiple times – both in the InMemory store and in the later EclipseStore implementation. This redundancy not only led to maintenance costs but also made uniform error handling more difficult. The solution to this problem was to introduce a dedicated component, the MappingCreator, which acts as a central element between validation, alias policy, and persistence mediation.\nThe MappingCreator covers the entire process chain from the alias check to the generation of a new short code to the transfer to persistent storage. This component follows the principle of functional composition and is designed to operate independently of the specific storage technology. This means that both the InMemory and EclipseStore approaches can use the exact generation mechanism without duplicating the logic.\nThe class is fully implemented in Core Java and does not require any external libraries. It defines a functional interface, ExistsByCode, that checks whether a specific shortcode already exists, and another interface, PutMapping, that stores a newly created mapping. This makes the Creator a generic tool that can take advantage of any implementation of UrlMappingStore . The following code snippet illustrates the structure:\npublic final class MappingCreator implements HasLogger { private final ShortCodeGenerator generator; private final ExistsByCode exists; private final PutMapping store; private final clock clock; private final Function\u0026lt;ErrorInfo, String\u0026gt; errorMapper; public MappingCreator(ShortCodeGenerator generator, ExistsByCode exists, PutMapping store, Clock clock, Function\u0026lt;ErrorInfo, String\u0026gt; errorMapper) { this.generator = Objects.requireNonNull(generator); this.exists = Objects.requireNonNull(exists); this.store = Objects.requireNonNull(store); this.clock = Objects.requireNonNullElse(clock, Clock.systemUTC()); this.errorMapper = Objects.requireNonNull(errorMapper); } The complete creation process is then displayed in the create method. The process follows a precisely defined scheme: First, it is checked whether the user has entered an alias. If so, it is checked and normalised via the AliasPolicy\u0026rsquo;s validate method. If the alias is invalid, the creator creates an error object with HTTP and application code and returns it as a failure result. If no alias is available, the creator automatically generates a unique shortcode with the ShortCodeGenerator, checks for collisions, and repeats the process if necessary.\npublic Result\u0026lt;ShortUrlMapping\u0026gt; create(String alias, String url) { logger().info(\u0026#34;createMapping - alias=\u0026#39;{}\u0026#39; / url=\u0026#39;{}\u0026#39;\u0026#34;, alias, url); final String shortCode; if (!isNullOrBlank(alias)) { var aliasCheck = AliasPolicy.validate(alias); if (aliasCheck.failed()) { var reason = aliasCheck.reason(); var reasonCode = switch (reason) { case NULL_OR_BLANK -\u0026gt; \u0026#34;ALIAS_EMPTY\u0026#34;; case TOO_SHORT -\u0026gt; \u0026#34;ALIAS_TOO_SHORT\u0026#34;; case TOO_LONG -\u0026gt; \u0026#34;ALIAS_TOO_LONG\u0026#34;; case INVALID_CHARS -\u0026gt; \u0026#34;ALIAS_INVALID_CHARS\u0026#34;; case RESERVED -\u0026gt; \u0026#34;ALIAS_RESERVED\u0026#34;; }; var errorJson = errorMapper.apply(new ErrorInfo(\u0026#34;400\u0026#34;, reason.defaultMessage, reasonCode)); return Result.failure(errorJson); } var normalized = normalize(alias); if (exists.test(normalized)) { var errorJson = errorMapper.apply(new ErrorInfo(\u0026#34;409\u0026#34;, \u0026#34;normalizedAlias already in use\u0026#34;, \u0026#34;ALIAS_CONFLICT\u0026#34;)); return Result.failure(errorJson); } shortCode = normalized; } else { String gen = normalize(generator.nextCode()); while (exists.test(gen)) { gen = normalize(generator.nextCode()); } shortCode = gen; } var mapping = new ShortUrlMapping(shortCode, url, Instant.now(clock), Optional.empty()); store.accept(mapping); return Result.success(mapping); } This method shows the functional structure of the component as an example. The creator does not create side effects outside its intended interfaces. It communicates exclusively via the function objects exists and store. Error handling is fully integrated into the result tree, allowing a clean separation between success and failure paths. In this way, logic remains deterministic, testable, and reproducible.\nWith the introduction of MappingCreator, the codebase has been significantly streamlined. Both the InMemoryUrlMappingStore and the EclipseStoreUrlMappingStore now use the same generation logic, which substantially improves consistency and extensibility. The Creator is thus not only a refactoring result, but also a strategic architectural element that combines functional abstraction, type safety and test-driven development.\nEclipseStore – The Persistent Foundation # After the internal generation logic has been unified with the MappingCreator, the implementation of the EclipseStoreUrlMappingStore now serves as the basis for permanent storage of the generated short URLs. This class replaces volatile storage with a fully persistent object model that maintains the application\u0026rsquo;s state across reboots. While the InMemory store only kept all mappings in a ConcurrentHashMap, in the EclipseStore they are stored within a serializable object, the so-called DataRoot. This structure serves as an anchor for all stored data elements, ensuring the entire object graph remains consistent at all times.\nThe central idea of the EclipseStore approach is to implement storage logic not via relational mappings but via object-oriented persistence. This preserves the application\u0026rsquo;s model in its entirety, without requiring object or database table conversions. The following example shows the definition of the DataRoot:\npublic class DataRoot implements Serializable { @Serial private static final long serialVersionUID = 1L; private final Map\u0026lt;String, ShortUrlMapping\u0026gt; mappings = new ConcurrentHashMap\u0026lt;\u0026gt;(); public Map\u0026lt;String, ShortUrlMapping\u0026gt; mappings() { return mappings; } } In the EclipseStoreUrlMappingStore, when the server starts, it checks whether a root structure already exists. If not, it is recreated and registered as the root node of the memory instance. The following snippet shows the relevant section from the constructor:\nvar storagePath = Paths.get(storageDir); Files.createDirectories(storagePath); this.storage = EmbeddedStorage.start(storagePath); DataRoot r = (DataRoot) storage.root(); if (r == null) { storage.setRoot(new DataRoot()); storage.storeRoot(); } This initialisation ensures that the application has the same persisted state both during a cold boot and after a shutdown. All access to the mappings then takes place directly via the root object. When a new mapping is created, the store calls the storeMappingAndPersist method , which stores the object in the internal map and synchronises the memory.\nprivate void storeMappingAndPersist(ShortUrlMapping m) { var dataRoot = (DataRoot) storage.root(); var mappings = dataRoot.mappings(); mappings.put(m.shortCode(), m); storage.store(mappings); } This immediate write process saves changes immediately. The EclipseStore automatically takes over transaction management at the object level and ensures atomic storage without requiring additional commit logic. This reduces the risk of errors and increases the traceability of the system’s status.\nIn addition, the EclipseStoreUrlMappingStore implements all the methods of the UrlMappingStore interface, including find, delete, count, and existsByCode. These methods operate directly on the object graph and use helper classes, such as UrlMappingFilterHelper, to efficiently execute queries and perform sorting. The key difference to the InMemory store is that changes to the data structure take effect immediately in persistent memory and are therefore available again when the application restarts.\nThe implementation of the EclipseStoreUrlMappingStore makes it clear that persistence was not designed as an afterthought, but as an integral part of the architecture. The entire data flow – from alias generation to MappingCreator to storage in DataRoot – follows a consistent design. This not only achieves persistence, but also anchors it conceptually: the memory is not an external component, but part of the living object model of the application.\nAdditional improvements in the core # Parallel to the introduction of the EclipseStore and the standardisation of the mapping logic, the central helper classes in the core module have also been further developed to ensure more consistent data processing. The focus was not on expanding the range of functions, but on making internal processes more precise. In particular, JsonUtils, UrlMappingFilterHelper, and the internal validation logic of the AliasPolicy have been explicitly revised to improve integration between the REST layer, the UI, and memory.\nThe JsonUtils class has been fully refactored to improve the security and robustness of data object serialisation and deserialisation. Instead of resorting to external parsers or frameworks, the class implements a well-defined strategy for simple objects and for nested structures. The focus is on stability, readability and deterministic processing. A central component is the toJsonListingPaped method, which is used for tabular representation in the administration interface. It generates a JSON array from a list of mapping objects, including metadata such as the total number and paging information.\npublic static String toJsonListingPaged(List\u0026lt;ShortUrlMapping\u0026gt; list, int total) { var sb = new StringBuilder(); sb.append(\u0026#39;{\u0026#39;); sb.append(\u0026#34;\\\u0026#34;total\\\u0026#34;:\u0026#34;).append(total).append(\u0026#39;,\u0026#39;); sb.append(\u0026#34;\\\u0026#34;items\\\u0026#34;:[\u0026#34;); for (int i = 0; i \u0026lt; list.size(); i++) { sb.append(toJson(list.get(i))); if (i \u0026lt; list.size() - 1) sb.append(\u0026#39;,\u0026#39;); } sb.append(\u0026#34;]}\u0026#34;); return sb.toString(); } This method exemplifies the class\u0026rsquo;s philosophy: complete control over serialisation and field order. This guarantees that the generated JSON structures correspond precisely to the expected format – an important point when the backend and UI are developed independently of each other. The use of StringBuilder is not a stylistic device, but a deliberate design to minimise garbage objects under high request load.\nAnother central component is the UrlMappingFilterHelper, which handles dynamic processing of filter and sorting parameters for REST endpoints. This class abstracts the transformation of the user-entered filters into internal predicate objects, which are then applied to the stored mappings at runtime. The implementation supports flexible queries without requiring specialised query languages or parsers. The following excerpt shows the method that evaluates a URL mapping based on several parameters:\npublic static boolean matches(ShortUrlMapping m, UrlMappingListRequest req) { if (req.codePart() != null \u0026amp;\u0026amp; !m.shortCode().contains(req.codePart())) return false; if (req.urlPart() != null \u0026amp;\u0026amp; !m.originalUrl().contains(req.urlPart())) return false; if (req.from() != null \u0026amp;\u0026amp; m.createdAt().isBefore(req.from())) return false; if (req.to() != null \u0026amp;\u0026amp; m.createdAt().isAfter(req.to())) return false; return true; } This compact logic ensures that filter requests are processed directly at the object level. The system does not have to generate intermediate representations or convert data. The result is efficient, storage-level filtering that scales to large datasets. In combination with the store\u0026rsquo;s advanced counting methods, it provides a high-performance foundation for complex search and administrative functions.\nThis area is rounded off by the extension of the AliasPolicy, in particular by the introduction of the helper method failed(). This is used for precise error evaluation and simplifies the control logic in components such as the MappingCreator. Instead of laboriously checking validation results, the developer can directly ask if a validation has failed, and then take fine-grained action based on the return value. This not only improves the readability of the codebase but also the overall system\u0026rsquo;s error resistance.\nThe adjustments above make it clear that there is a well-thought-out concept behind the supposed auxiliary classes. They are not marginal components, but supporting pillars of data consistency and system stability. Through targeted optimisations and a clear methodological framework, a core module is created that serves as the basis for the project\u0026rsquo;s further development, both functionally and architecturally.\nBefore \u0026amp; After – Impact on the developer experience # With the integration of EclipseStore and the unification of generation logic via MappingCreator, the developer experience in this project has fundamentally changed. What began as a loose interplay of individual components has developed into a precisely orchestrated system that convinces both in its architecture and maintainability. This is particularly evident in the reduction of redundancies, improved testability and increased transparency in the code flow.\nIn the past, central operations – especially the creation and verification of new mappings – were scattered across multiple classes. The InMemory store contained its own validation and checking routines, while later extensions had to reimplement the same logic for persistent storage. With the MappingCreator, this process has now been transformed into a clearly defined, reusable unit. This not only eliminates duplicate code but also the risk of different implementations delivering divergent results. The following excerpt shows an example of how the creation of a mapping was previously realised within the store:\npublic ShortUrlMapping create(String alias, String url) { String shortCode = alias != null ? alias : generator.nextCode(); if (map.containsKey(shortCode)) { throw new IllegalStateException(\u0026#34;Alias already exists: \u0026#34; + shortCode); } var mapping = new ShortUrlMapping(shortCode, url, Instant.now(), Optional.empty()); map.put(shortCode, mapping); return mapping; } This logic was functionally correct, but inflexible. Neither validation nor error objects nor alias rules were built in, and there was no uniform handling of failures or conflicts. With the introduction of MappingCreators and the centralised error structure, this process has been redesigned. The result is a deterministic, controlled, and traceable generation of short URLs, in which every step – from input to persistence – occurs via clearly defined interfaces.\nThe second significant difference concerns how the memory is addressed. Previously, the entire application was designed for volatile data retention, which meant that each reboot resulted in a complete loss of the stored mappings. With the EclipseStore, this state has been fundamentally changed. The following code shows how the Store initialises its persistent database when the application starts:\nvar storagePath = Paths.get(storageDir); Files.createDirectories(storagePath); this.storage = EmbeddedStorage.start(storagePath); DataRoot r = (DataRoot) storage.root(); if (r == null) { storage.setRoot(new DataRoot()); storage.storeRoot(); } These few lines mark the crucial difference between temporary and permanent system states. The DataRoot\u0026rsquo;s data structure is loaded at startup, changes are immediately synchronised, and the state is maintained across reboots. From a developer\u0026rsquo;s point of view, this means that unit tests, integration tests and runtime behaviour can now consistently access the same database. The difference between development and production disappears because both use the same data flow and persistence logic.\nThis standardisation also directly affects the user interface. The Vaadin components, such as OverviewView and StoreIndicator, now obtain their data via clearly defined interfaces that are decoupled from the persistence layer. State changes of the store or new mappings trigger events that are consumed by the UI components, without explicit dependencies. This creates a consistent, reactive system that works coherently from the data source to the surface.\nThis development has fundamentally changed how developers interact with code. Bugs are easier to reproduce, tests are more stable, and extensions can be modular. The code is no longer a collection of independent methods, but a structured system in which each component has its place and responsibility.\nCheers Sven\n","date":"5 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-persistence-part-02/","section":"Posts","summary":"Today, we will finally integrate the StoreIndicator into the UI.\nVaadin integration: live status of the store Implementation of the StoreIndicator Refactoring inside – The MappingCreator as a central logic. EclipseStore – The Persistent Foundation Additional improvements in the core Before \u0026 After – Impact on the developer experience The source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-02\n","title":"Advent Calendar - 2025 - Persistence – Part 02","type":"posts"},{"content":"**Visible change: When the UI shows the memory state\nWith the end of the last day of the Advent calendar, our URL shortener was fully functional: the admin interface could filter, sort, and display data page by page – performant, cleanly typed, and fully implemented in Core Java. But behind this surface lurked an invisible problem: All data existed only in memory. As soon as the server was restarted, the entire database was lost.\nThe source code for this version can be found on GitHub athttps://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-02\nHere\u0026rsquo;s the screenshot of the version we\u0026rsquo;re going to implement now.\n**Why pure in-memory data is reaching its limits\nThis volatility was acceptable for early testing, but as the functionality depth increased, clear limits became apparent. In-memory data has three fundamental drawbacks:\nVolatility: Each reboot resets the state – a behaviour that leads to unusable test and production data in real admin systems. Missing history: Statistical evaluations or comparisons across sessions are impossible because the state is not persisted over time. Invisible state: The user does not see in the UI whether they are working with persistent or volatile memory. This meant that the application remained technically correct, but operationally incomplete. A real administration interface not only needs functions, but also reliability and transparency about the system status.\n**Motivation for the introduction of EclipseStore\nThis is where EclipseStore comes into play.The project aims to make a seamless transition from in-memory data structures to persistent object graph storage – without ORM, without database, without a break in the architecture.\nThis makes it perfectly in line with the philosophy of this project: type safety, simplicity and complete control over the data flow.\nEclipseStore stores the same object graph that we already use in memory permanently on disk.No mapping, no SQL, no external dependency injection layer – the application remains pure Java. By integrating EclipseStoreUrlMappingStore, the shortener can run in two modes:\nInMemory for quick tests and volatile sessions, EclipseStore for productive, persistent runtimes. This duality not only enables real operational data, but also a clean transition between the development and production environments – an essential aspect if you want to systematically upgrade applications without rewriting them.\n**Goal: Permanent storage and live status in the UI\nBut persistence alone was not the goal of this chapter; it was at least as necessary that the condition became visible. If the server is in persistent mode, the user interface should show this immediately – preferably in real time.\nThis is precisely where the new status display in the MainLayout comes in: A small icon with colour feedback (green, blue, red) signals whether the system is currently volatile, persistent or unreachable. Underneath are mechanisms that cyclically query the server state, distribute events, and automatically update the UI when the mode changes.\nThis merges the backend and frontend into a standard, reactive system: The interface no longer becomes just a window on stored data, but a visible mirror of the system state.\nThis coupling of persistence and UI status forms the foundation for all upcoming enhancements – especially for security, rights management and operational diagnostics.\n**The new MainLayout – context and status at a glance\nAfter we have created the theoretical basis in the previous chapter and understood why volatile InMemory data is not sufficient and how EclipseStore enables persistence, we now turn our attention to the user interface. Every architectural decision becomes tangible only when it is reflected in the UI, both visually and functionally.\nThe most visible expression of this change is in the MainLayout. With the introduction of EclipseStore, it received a new, subtle, but highly effective addition: the StoreIndicator. This small but central component is located in the application header and shows at a glance whether the shortener is currently using a volatile InMemory store or a persistent EclipseStore.\nThis transforms a previously static header into a dynamic control centre that is not only navigative, but also informative – a visual link between the user interface and the system state.\n**Architectural Concept – Visibility as a System Function\nIn the original Vaadin interface, the header layout consisted only of a title and the navigation structure. The new component now expands this area: between the title \u0026ldquo;URL Shortener\u0026rdquo; and the menu, an icon with a label appears, whose colour and text change depending on the memory status.\nThe concept behind it is deliberately kept minimalist: Status information should not be displayed via dialogues or messages, but should be permanently present , similar to a status bar in professional administration tools.\n**Technical integration\nThe integration was carried out via the existing MainLayout, which serves as the central layout template in Vaadin. There, a StoreIndicator is added when building the header :\nvar adminClient = AdminClientFactory.newInstance(); var storeIndicator = new StoreIndicator(adminClient); storeIndicator.getStyle().set(\u0026#34;margin-left\u0026#34;, \u0026#34;auto\u0026#34;); Align Right HorizontalLayout headerRow = new HorizontalLayout(toggle, appTitle, new Span(), storeIndicator); headerRow.setWidthFull(); headerRow.setAlignItems(FlexComponent.Alignment.CENTER); The StoreIndicator internally uses an AdminClient that periodically queries the server endpoint /store/info. This provides JSON information about the current mode, the number of saved mappings, and the time of server start.\n**UI design and colour code\nFor better orientation, a three-level colour scheme has been introduced:\n🟢 EclipseStore active: Persistent persistence, data is written to disk. 🔵 In-Memory Mode: Volatile memory, suitable for testing and fast development cycles. 🔴 Error condition: Connection unreachable or response invalid. This colour system is implemented consistently with Vaadin\u0026rsquo;s Lumo design system and ensures that the status indicator blends harmoniously into the UI.\n**Effect on user experience\nThe effect is subtle but profound:\nUsers can immediately recognise whether they are working with a persistent or a temporary system. At the same time, there is greater trust in the application, as the interface now provides transparency into the system’s status. This aspect is often neglected in professional web applications.\nIn the next chapter, we will look at how the StoreIndicator works technically: from the polling mechanism to the event architecture to its self-updating in the UI.\n**Live Status Component: The StoreIndicator in Detail\nAfter the visible integration in the MainLayout, let\u0026rsquo;s now turn our attention to the StoreIndicator\u0026rsquo;s actual functionality**.** This central component displays the system’s state in real time. While the header in the UI provides the context, the StoreIndicator is the heart that makes data movement visible and gives immediate feedback to the user.\n**Architecture and Purpose\nThe StoreIndicator was designed as a standalone Vaadin component that extends HorizontalLayout. It combines logic, display and event control in a clearly defined unit. Their goal: to regularly retrieve the current memory state (InMemory, EclipseStore or error state) and display it visually.\nTo do this, the component uses periodic polling, which queries the server endpoint /store/info via an AdminClient . The response includes metadata such as memory mode, number of saved mappings, and the time of server startup. Based on this data, the display changes dynamically.\n**Lifecycle in the Vaadin UI\nWhen the user interface is attached (onAttach), polling is activated. The UI retrieves the current system information at fixed intervals (every 10 seconds). When removing (onDetach), this background activity is terminated again, keeping the application resource-efficient.\n@Override protected void onAttach(AttachEvent attachEvent) { refreshOnce(); UI ui = attachEvent.getUI(); ui.setPollInterval(10000); every 10 seconds ui.addPollListener(e -\u0026gt; refreshOnce()); } **Communication with the backend\nCommunication between the frontend and the backend occurs via the AdminClient. This calls the JSON endpoint /store/info and converts the response to a StoreInfo object:\n**StoreInfo info = adminClient.getStoreInfo();\n**boolean persistent = \u0026ldquo;EclipseStore\u0026rdquo;.equalsIgnoreCase(info.mode());\nOn this basis, color, icon and text are updated. The visual status is based on three states:\n🟢 EclipseStore active – persistent storage, data is stored on disks. 🔵 InMemory mode – volatile storage, ideal for testing and development. 🔴 Error condition – Connection lost or server unreachable. These states are immediately visible in the UI and follow the colour values of Vaadin\u0026rsquo;s Lumo theme, which ensures a consistent design.\n**Event-Based Update\nIn addition to polling, the StoreIndicator relies on an event-based notification system. Via StoreEvents.publish(), it sends a global event whenever the storage state changes. Components such as the OverviewView can subscribe to this event and automatically reload when changes are made.\n**StoreEvents.publish(new StoreConnectionChanged(newMode, info.mappings()));\nThis loosely coupled event system avoids direct dependencies between UI components and allows for flexible extensibility – a pattern that can also be applied to other UI areas.\nThe StoreIndicator, which displays the system\u0026rsquo;s state in real time, operates resource-efficiently and enhances user confidence in the application. It turns the abstract term \u0026ldquo;persistence mode\u0026rdquo; into a visual experience – a perfect example of how backend information can be elegantly integrated into the UI.\nCheers Sven\n","date":"4 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-persistence-part-01/","section":"Posts","summary":"**Visible change: When the UI shows the memory state\nWith the end of the last day of the Advent calendar, our URL shortener was fully functional: the admin interface could filter, sort, and display data page by page – performant, cleanly typed, and fully implemented in Core Java. But behind this surface lurked an invisible problem: All data existed only in memory. As soon as the server was restarted, the entire database was lost.\n","title":"Advent Calendar - 2025 - Persistence – Part 01","type":"posts"},{"content":"In the previous part, we looked at the implementation on the server side. This part is now about the illustration on the user page.\nThe source code for the initial state can be found on GitHub under https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-00 .\nThe following screenshot shows this state of development.\nThe focus of this Advent calendar day is on the introduction of targeted filtering, search and paging functionality on the UI side. The goal is to extend the existing \u0026ldquo;Overview\u0026rdquo; view so that users can search specifically for specific shortcodes or URL fragments, restrict time periods and retrieve the results page by page. All the technical foundations for this have been laid in the previous part.\nThe source code for today\u0026rsquo;s articles can be found on Github under https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-01 .\nThe following screenshot shows the state of development that we will reach today.\nIntegration into the Vaadin UI # After implementing the filtering and paging logic on the server and client side, the focus is now on integration into the user interface. The Vaadin UI acts as the visible layer of the system and is crucial to the user experience. The aim of this chapter is to show how the new functions – filtering, searching and page by page – are brought together in a modern, reactive interface.\nIn contrast to the previous parts, which primarily described technical structures, the focus here is on the interaction between user and system. The OverviewView serves as a central view through which users communicate directly with the API without having to know technical details.\nThe following sections show how the new API endpoints and client methods create a dynamic yet intuitive interface – from data binding and filter logic to visual fine-tuning and ergonomic design.\nOverviewView – Filtering, Searching, and Paging in the Vaadin UI # With the new server and client functions, the basis for an interactive user interface is now available. This chapter describes the revised OverviewView – the central view of the Vaadin application in which all existing short links can be listed, filtered and managed.\nThe original version of the View only showed a complete list of all mappings. In the new version, it has been expanded into a dynamic, filterable and page-based interface that communicatesdirectly with the endpoints/listand/list/count.\nStructure of the user interface # The OverviewView combines several input elements to capture search and filter criteria:\nText fields for shortcode and URL substrings (codePart, urlPart) Case-sensitive checkboxes (codeCase, urlCase) Date and time selection for time periods (fromDate, toDate) Dropdowns for sorting (sortBy, dir) Page size via a numeric input field (pageSize) In addition, navigation buttons (Prev / Next) enable side control. All input is transferred to a UrlMappingListRequest object, which is passed to the URLShortenerClient when the view is refreshed .\nCentral logic: DataProvider # The data supply to the grid is provided by a CallbackDataProvider, which reacts dynamically to user interactions. This calls two methods of the client in the background:\n**list()** – loads the actual records of the current page. **listCount()** – determines the total number to control page navigation. A simplified excerpt of the initialization:\ndataProvider = new CallbackDataProvider\u0026lt;\u0026gt;( q -\u0026gt; { var req = buildFilter(currentPage, pageSize.getValue()); return urlShortenerClient.list(req).stream(); }, q -\u0026gt; { var baseReq = buildFilter(null, null); totalCount = urlShortenerClient.listCount(baseReq); refreshPageInfo(); return totalCount; } ); This architecture enables the View to react immediately to any change in the filter without having to reload the entire page. The user can conveniently browse through the results, adjust filters or change time periods – the data is loaded live in each case.\nUX Improvements # In addition to the functional enhancement, the user experience has also been improved:\nResponsive layouts: All filter elements arrange themselves flexibly, even with smaller window sizes. Automatic page navigation: The Prev and Next buttons are automatically deactivated depending on the context. Date-time combination: Separate DatePicker and TimePicker fields keep input precise and intuitive. These innovations lead to considerable added value in the daily use of the application: Instead of static tables, there is now an interactive, reactive interface that is both more performant and more ergonomic.\nIn the next section, we\u0026rsquo;ll look at the implementation of the paging and filtering logic in detail, including the buildFilter() method and how to handle the parameters in the UI.\nImplementation details of the filter logic – buildFilter() and data binding # The buildFilter() method forms the heart of the new filter mechanics in the Vaadin interface. It collects all of the user\u0026rsquo;s input—such as text fields, checkboxes, dates, and sorting options—and transfers them to a UrlMappingListRequest object. This request object is then passed on to the client, which uses it to generate the appropriate query parameters for the server.\nStructure of the buildFilter() method # The focus is on the builder structure of the UrlMappingListRequest. The method first reads all UI components, checks for empty fields, and creates a consistent filter object from them:\nprivate UrlMappingListRequest buildFilter(Integer page, Integer size) { UrlMappingListRequest.Builder b = UrlMappingListRequest.builder(); if (codePart.getValue() != null \u0026amp;\u0026amp; !codePart.getValue().isBlank()) { b.codePart(codePart.getValue()); } b.codeCaseSensitive(Boolean.TRUE.equals(codeCase.getValue())); if (urlPart.getValue() != null \u0026amp;\u0026amp; !urlPart.getValue().isBlank()) { b.urlPart(urlPart.getValue()); } b.urlCaseSensitive(Boolean.TRUE.equals(urlCase.getValue())); if (fromDate.getValue() != null \u0026amp;\u0026amp; fromTime.getValue() != null) { var zdt = ZonedDateTime.of(fromDate.getValue(), fromTime.getValue(), ZoneId.systemDefault()); b.from(zdt.toInstant()); } else if (fromDate.getValue() != null) { var zdt = fromDate.getValue().atStartOfDay(ZoneId.systemDefault()); b.from(zdt.toInstant()); } if (toDate.getValue() != null \u0026amp;\u0026amp; toTime.getValue() != null) { var zdt = ZonedDateTime.of(toDate.getValue(), toTime.getValue(), ZoneId.systemDefault()); b.to(zdt.toInstant()); } else if (toDate.getValue() != null) { var zdt = toDate.getValue().atTime(23, 59).atZone(ZoneId.systemDefault()); b.to(zdt.toInstant()); } if (sortBy.getValue() != null \u0026amp;\u0026amp; !sortBy.getValue().isBlank()) b.sort(sortBy.getValue()); if (dir.getValue() != null \u0026amp;\u0026amp; !dir.getValue().isBlank()) b.dir(dir.getValue()); if (page != null \u0026amp;\u0026amp; size != null) { b.page(page).size(size); } return b.build(); } The method works strictly defensively: empty fields are ignored, optional values are only set if they actually exist. The result is always a valid, well-defined request object.\nData flow between UI and server # The generated filter objects are passed to theURLShortenerClient via the CallbackDataProvider. This generates the query string for the HTTP request. This creates a clear, linear chain:\nUI → UrlMappingListRequest → client → /list endpoint → filter → results Every change in the UI – be it a new search, a changed page size or a date filter – automatically triggers a refresh. The Grid component then updates itself with the newly filtered data.\nAdvantages of encapsulation # This structure offers several advantages:\nReusability:buildFilter() encapsulates all UI logic in one method. Testability: The result can be easily tested in unit tests without the need for Vaadin-specific classes. Extensibility: New filter fields can be easily added as long as they are stored in the builder. This clean decoupling keeps the Vaadin interface clutter-free, while at the same time achieving deep integration with the API. The buildFilter() method is thus the central node between user interaction and server-side data logic.\nPaging and user interaction – control, display, feedback # The revised OverviewView offers intuitive control for page-by-page navigation through large result sets. The aim is for users to quickly see where they are in the results list, how many elements exist in total and which interactions are currently possible.\nPaging elements in the UI # Buttons:Prev (to the previous page) and Next (to the next page). Page size:IntegerField pageSize with min/max limits and step buttons. Status display:pageInfo shows e.g. \u0026ldquo;Page 3 / 12 • 289 total\u0026rdquo;. These elements are directly linked to the DataProvider and are updated after each query.\nChanging sides and limiting # Clicking on Prev/Next adjusts the current page, then the View reloads the data:\nprevBtn.addClickListener(e -\u0026gt; { if (currentPage \u0026gt; 1) { currentPage--; refresh(); } }); nextBtn.addClickListener(e -\u0026gt; { int size = Optional.ofNullable(pageSize.getValue()).orElse(25); int maxPage = Math.max(1, (int) Math.ceil((double) totalCount / size)); if (currentPage \u0026lt; maxPage) { currentPage++; refresh(); } }); The page size directly affects the calculation of maxPage. Changes to the pageSize field reset the current page to 1 and trigger a refresh:\npageSize.addValueChangeListener(e -\u0026gt; { currentPage = 1; grid.setPageSize(Optional.ofNullable(e.getValue()).orElse(25)); refresh(); }); Status Indicator Update # The refreshPageInfo() method synchronizes buttons and display with the current data state:\nprivate void refreshPageInfo() { int size = Optional.ofNullable(pageSize.getValue()).orElse(25); int maxPage = Math.max(1, (int) Math.ceil((double) totalCount / size)); currentPage = Math.min(Math.max(1, currentPage), maxPage); pageInfo.setText(\u0026#34;Page \u0026#34; + currentPage + \u0026#34; / \u0026#34; + maxPage + \u0026#34; • \u0026#34; + totalCount + \u0026#34; total\u0026#34;); prevBtn.setEnabled(currentPage \u0026gt; 1); nextBtn.setEnabled(currentPage \u0026lt; maxPage); } This prevents the navigation from getting into invalid states (e.g. Prev on page 1 or Next after the last page).\nInteraction with list() and listCount() # Count (**listCount**): Determines the total amount (totalCount) without transferring data that does not need to be displayed. Data (**list**): Loads only the currently visible subset (based on page/size and current filters). The combination minimizes network load and ensures consistently fast response times.\nUX details and fault tolerance # Disable logic: Buttons are automatically (de)activated depending on currentPage/maxPage. Error handling: Network or server errors are immediately visible viaNotification.show(\u0026ldquo;Loading failed\u0026rdquo;). Reset behavior: The reset button sets filters to default values, currentPage to 1 and triggers refresh(). These mechanisms create a reactive, robust paging experience that remains clear even with very large amounts of data and provides clear feedback to the user.\nUX \u0026amp; styling – visual and ergonomic refinements # Now that the functionality of the filtering, search and paging mechanisms has been established, this chapter focuses on the fine-tuning of the user interface. The aim is to improve the user experience of the Vaadin interface without changing the technical structure. The focus is on clarity, responsiveness and consistent visual feedback.\nBasic design principles # Clarity over complexity – Each function (e.g., filtering, sorting, paging) should be visually recognizable without overwhelming the user with options. Visual grouping – Logically related items are grouped into horizontal or vertical layouts (e.g., time range, sorting options, search fields). Consistent feedback – Every user action (e.g., changing a filter, changing pages, deleting action) receives direct visual or textual feedback. Exemplary adjustments # 1. Layout structure # The input elements for filters and paging have been divided into several layout lines:\nvar filterRow = new HorizontalLayout( codePart, codeCase, urlPart, urlCase, fromGroup, toGroup, sortBy, dir ); filterRow.setDefaultVerticalComponentAlignment(Alignment.END); var pagingRow = new HorizontalLayout(prevBtn, nextBtn, pageInfo, pageSize); pagingRow.setDefaultVerticalComponentAlignment(Alignment.CENTER); add(filterRow, pagingRow, grid); This division semantically separates filters and navigation and thus improves visual orientation.\n2. Color and Theme Consistency # The buttons now use Vaadin\u0026rsquo;s own theme variants to convey semantic meaning:\nsearchBtn.addThemeVariants(ButtonVariant.LUMO_PRIMARY); resetBtn.addThemeVariants(ButtonVariant.LUMO_CONTRAST); deleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR, ButtonVariant.LUMO_TERTIARY); This immediately signals to the user which actions are primary, neutral or destructive.\n3. Responsive behavior # By using setWrap(true) and percentage widths, the layout is also optimally displayed on smaller screens. Filter elements automatically wrap into new lines without disturbing the overall impression.\nfilterRow.setWrap(true); filterRow.setWidthFull(); 4. Interactive status messages # Error and success messages are communicated via Notification.show(). In addition, the affected element can optionally be visually marked – for example, by means of temporary color changes or tooltips.\nNotification.show(\u0026#34;Short link deleted.\u0026#34;); Bringing together function and aesthetics # These design and ergonomic fine-tunes make the OverviewView a reactive tool for everyday use. The application not only appears technically sophisticated, but also visually conveys the demand for precision and quality.\nThis completes the implementation of the UI for the new filter and search logic – a basis on which later extensions such as caching, auto-suggest or extended sorting logics can be built.\nConclusion and outlook # With the implementation of the filter, search and paging functions, the first milestone of the Advent calendar project has been reached. A static overview has become a dynamic, interactive system that allows data to be accessed in a targeted and efficient manner. The basic idea of the project – a solution implemented entirely in Core Java without external frameworks – has been consistently retained.\nReview # This day showed how a clear separation of responsibilities and clean API design can create a solid basis for more complex functionalities:\nServer-side : Introduction of the UrlMappingFilter and expansion of the InMemoryUrlMappingStore for structured queries. Client-side : Extension of the URLShortenerClient with type-safe methods for filtering and counting. UI side : Integration of the new features into the Vaadin-based interface with reactive paging, intuitive navigation and clear user guidance. The result is a consistent overall system that remains modular, expandable and testable.\nCheers Sven\n","date":"3 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-filter-search-part-02/","section":"Posts","summary":"In the previous part, we looked at the implementation on the server side. This part is now about the illustration on the user page.\nThe source code for the initial state can be found on GitHub under https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-00 .\n","title":"Advent Calendar - 2025 - Filter \u0026amp; Search – Part 02","type":"posts"},{"content":"With the Vaadin interface described in [Part III], our URL shortener has a fully functional administration console available for the first time. It allows viewing existing short links in tabular form and managing them manually. But after just a few dozen entries, a clear limit becomes apparent: displaying all saved mappings is neither performant nor user-friendly. An efficient shortener must be able to scale – not only when generating, but also when searching through its data.\nArchitecture Extension Server-side changes UrlMappingFilter – the new filter model ListHandler - Extended GET Endpoint (/list) ListCountHandler - Lightweight counting endpoint (/list/count) InMemoryUrlMappingStore – Filtering, sorting, and paginating QueryUtils – Parsing and Normalising Query Parameters Introduction to API Endpoints Structure of the new endpoints Compatibility and stability Supported Parameters Interaction of the parameters Client-side extensions UrlMappingListRequest – Filter and Paging Request Builder Extension of the URLShortenerClient – Filter and Count New methods Significance of enlargement Backward compatibility Client Test Coverage – URLShortenerClientListTest Paging and sorting tests Aim of the tests The source code for this status can be found on GitHub at https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-00.\nThe screenshot below shows the status we start with.\nThe focus of this first day of the Advent calendar is therefore on introducing targeted filtering, search, and paging functionality. The goal is to extend the existing \u0026ldquo;Overview\u0026rdquo; view so that users can search for specific shortcodes or URL fragments, restrict time periods, and retrieve results page by page. All the technical foundations for this have been laid in the previous parts:\nPart I – Short links, clear architecture : A detailed introduction to the motivation, architecture and data model of the project. The article shows how to develop a URL shortener entirely in Core Java, along with the design principles behind it. (https://svenruppert.com/2025/06/10/short-links-clear-architecture-a-url-shortener-in-core-java/ ) Part II – Deepening the Server Components : This article explores REST handlers, validation, security considerations, and the interaction between the core module and the server layer. (https://svenruppert.com/2025/06/20/part-ii-urlshortener-first-implementation/) Part III – The Web UI with Vaadin Flow: This shows how the user interface is structured, how lists, forms, dialogues and validations work, and how Vaadin Flow enables a modern UI without an additional JavaScript stack. (https://svenruppert.com/2025/08/15/part-iii-webui-with-vaadin-flow-for-the-url-shortener/) The source code for today\u0026rsquo;s article can be found on Github under https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-01.\nThe following screenshot shows the development status shown in it.\nIt is important that we do not introduce framework magic , but consistently expand the existing system with the JDK\u0026rsquo;s on-board resources. Every new class, additional parameter, and API detail follows the same principles as before: clarity, type safety, and transparency. The changes are deliberately incremental – they do not replace existing functions, but expand them in a controlled manner.\nBelow, we\u0026rsquo;ll take a step-by-step look at how this new layer was introduced: from the server-side filtering model to the extended REST endpoint and integration with the Vaadin UI. All new code sections are explicitly highlighted so that the differences to the previous parts can be clearly understood.\nArchitecture Extension # The module structure established in the previous parts is completely retained: core , api , client and ui-vaadin continue to form the central layers of the system. However, there is a new logical level between the REST endpoint and the data storage: the filter model. It allows you to describe what data the server is supposed to deliver precisely – and no longer just that it returns all existing mappings.\nThis architectural principle follows the responsibility separation principle described in [Part II]: Each layer should do precisely what is within its responsibility. The REST API receives requests, converts them into type-safe filter objects, and passes them to the store. The store, in turn, does the actual searching, sorting, and page layout. The UI only uses filtered results, which keeps it lightweight and responsive.\nThe new architecture can therefore be understood as a small but decisive extension of the previous data flow:\n[UI] → [Client] → [API] → [Filter Model] → [Store]\nPreviously, communication between the UI and the API was mainly in the form of complete data lists; now, queries with parameters are transmitted in a targeted manner. These parameters describe search patterns, time periods, sorting, and paging information. The result is a much more flexible and high-performance interaction that fits seamlessly into the existing system.\nConceptually, care was taken to ensure that all new components could be integrated incrementally. No existing function will be changed or replaced. Instead, clients that continue to use the legacy endpoints (e.g. /list/all) do so unchanged. New clients – such as the revised OverviewView – on the other hand, immediately benefit from the extended filter options.\nThis decoupling not only ensures stability during operation but also forms the basis for future extensions, such as persistent filters or server-side search indexes. The following sections now show in concrete terms how this architecture was technically implemented.\nServer-side changes # After the architecture has been defined stably in the previous parts, this section focuses on extending the server-side. The goal is to complement the existing REST endpoints so that they can deliver filtered, sorted, and paginated results in a targeted manner. The previous URL shortener structure has not changed, but has been expanded into clearly defined modules.\nThe following subchapters describe the main new building blocks – from the central UrlMappingFilter to the revised handlers and the helper classes that handle parsing and data processing.\nUrlMappingFilter – the new filter model # To enable targeted searches, a new class has been introduced: UrlMappingFilter. It forms the heart of the server-side extension and defines all the parameters that precisely describe a query – from text fragments and date ranges to sorting and paging.\nThe structure consistently follows the type-safe API construction principle established in [Part II]. Instead of evaluating unstructured query parameters directly in the handler, UrlMappingFilter encapsulates all possible filter options in a clearly defined object. This keeps the REST logic lean and the code easier to test.\nA typical filter object might look like this:\nvar filter = UrlMappingFilter.builder() .codePart(\u0026#34;ex-\u0026#34;) .urlPart(\u0026#34;docs\u0026#34;) .createdFrom(Instant.parse(\u0026#34;2025-10-01T00:00:00Z\u0026#34;)) .limit(25) .sortBy(SortBy.CREATED_AT) .direction(Direction.DESC) .build(); Here, only the parameters relevant to a specific query are provided as examples. Unset values remain null or empty, which makes the builder particularly flexible. The class itself is immutable – once it has been created, it cannot be changed.\nThe most important features at a glance:\ncodePart, urlPart: Text-based filtering (substrings, optional case-sensitive) createdFrom, createdTo: Time constraint of the result offset, limit: Paging control (start index and number of records) sortBy, direction: Sort criteria (e.g. CREATED_AT or SHORT_CODE) This clear structure allows the store to apply filters efficiently and easily integrate future extensions (such as status or user). It does not replace the existing search mechanism, but expands it modularly. Existing functions such as findAll() are retained.\nThe use of an explicit builder pattern ensures that filter objects can only be created in valid combinations. This principle has already proven itself with the ShortenRequest and ShortUrlMapping classes. In addition, consistent use of optional and explicit data types prevents empty strings or incorrect values from interfering with the filtering process.\nThis provides a basis for receiving requests in a structured format via the REST API and forwarding them to the store in a targeted manner. The following section shows how these filter objects are used within the ListHandler .\nListHandler - Extended GET Endpoint (/list) # The previous implementation of the ListHandler in [Part II] primarily returned all stored mappings as a static list. As the data stock grew, this was neither performant nor differentiated enough. As part of the new filter architecture, the handler has therefore been fundamentally expanded –without changing its previous endpoints. Existing calls such as /list/all or /list/expired will continue to work unchanged.\nThe new code path now recognises requests to the /list endpoint and interprets the query parameters. These are transferred to a UrlMappingFilter object and then passed to the store. The store then returns the filtered and sorted subset.\nAn excerpt from the new method:\nprivate String listFiltered(HttpExchange exchange) { var query = parseQueryParams(exchange.getRequestURI().getRawQuery()); int page = parseIntOrDefault(first(query, \u0026#34;page\u0026#34;), 1); int size = clamp(parseIntOrDefault(first(query, \u0026#34;size\u0026#34;), 50), 1, 500); int offset = Math.max(0, (page - 1) * size); var sortBy = parseSort(first(query, \u0026#34;sort\u0026#34;)); var dir = parseDir(first(query, \u0026#34;dir\u0026#34;)); boolean codeCase = Boolean.parseBoolean(first(query, \u0026#34;codeCase\u0026#34;)); boolean urlCase = Boolean.parseBoolean(first(query, \u0026#34;urlCase\u0026#34;)); var filter = UrlMappingFilter.builder() .codePart(first(query, \u0026#34;code\u0026#34;)) .codeCaseSensitive(codeCase) .urlPart(first(query, \u0026#34;url\u0026#34;)) .urlCaseSensitive(urlCase) .createdFrom(parseInstant(first(query, \u0026#34;from\u0026#34;), true).orElse(null)) .createdTo(parseInstant(first(query, \u0026#34;to\u0026#34;), false).orElse(null)) .offset(offset) .limit(size) .sortBy(sortBy.orElse(null)) .direction(dir.orElse(null)) .build(); int total = store.count(filter); var results = store.find(filter); var items = results.stream().map(m -\u0026gt; toDto(m, Instant.now())).toList(); return JsonUtils.toJsonListingPaged(\u0026#34;filtered\u0026#34;, items.size(), items, page, size, total, sortBy.orElse(null), dir.orElse(null)); } This method illustrates how cleanly the new filter logic integrates into the existing retailer. Instead of returning a complete list, it now creates a paged response object that contains additional metadata:\n{ \u0026#34;mode\u0026#34;: \u0026#34;filtered\u0026#34;, \u0026#34;page\u0026#34;: 2, \u0026#34;size\u0026#34;: 25, \u0026#34;total\u0026#34;: 123, \u0026#34;sort\u0026#34;: \u0026#34;createdAt\u0026#34;, \u0026#34;dir\u0026#34;: \u0026#34;desc\u0026#34;, \u0026#34;count\u0026#34;: 25, \u0026#34;items\u0026#34;: [ { ... }, { ... } ] } This format offers two decisive advantages: On the one hand, the UI can work specifically with pagination, and on the other hand, API clients can be extended by further parameters in the future without breaking existing structures.\nThe trader must continue to pay attention to backward compatibility. All older endpoints are registered in the same context and continue to return complete JSON lists. Only /list uses the extended schema.\nThe ListHandler thus becomes the central intermediary between the API request and data filtering – it serves as the \u0026ldquo;translator\u0026rdquo; between HTTP parameters and the internal filter model. The following section shows how the supplemental counting endpoint /list/count uses the same logic for pure quantity queries.\nListCountHandler - Lightweight counting endpoint (/list/count) # In parallel with the extended /list endpoint, an additional handler, ListCountHandler, has been introduced. Its task is to provide only the total number of hits for a given filter configuration – without returning the actual data sets. This separation was deliberately chosen to make paging operations in the UI efficient.\nIn contrast to ListHandler, the counting endpoint does not return complete JSON objects with shortcodes and URLs, but only a compact counter value. This allows the client to determine in advance how many pages a query contains before loading individual data pages.\nAn excerpt from the implementation shows the simplicity of the approach:\n@Override public void handle(HttpExchange exchange) throws IOException { if (!\u0026#34; GET\u0026#34;.equalsIgnoreCase(exchange.getRequestMethod())) { exchange.sendResponseHeaders(405, -1); return; } Map\u0026lt;String, List\u0026lt;String\u0026gt;\u0026gt; q = parseQuery(Optional.ofNullable(exchange.getRequestURI().getRawQuery()).orElse(\u0026#34;\u0026#34;)); UrlMappingFilter filter = UrlMappingFilter.builder() .codePart(first(q, \u0026#34;code\u0026#34;)) .codeCaseSensitive(bool(q, \u0026#34;codeCase\u0026#34;)) .urlPart(first(q, \u0026#34;url\u0026#34;)) .urlCaseSensitive(bool(q, \u0026#34;urlCase\u0026#34;)) .createdFrom(parseInstant(first(q, \u0026#34;from\u0026#34;)).orElse(null)) .createdTo(parseInstant(first(q, \u0026#34;to\u0026#34;)).orElse(null)) .build(); int total = store.count(filter); byte[] body = (\u0026#34;{\\\u0026#34;total\\\u0026#34;:\u0026#34; + total + \u0026#34;}\u0026#34;).getBytes(StandardCharsets.UTF_8); exchange.getResponseHeaders().add(\u0026#34;Content-Type\u0026#34;, \u0026#34;application/json; charset=utf-8\u0026#34;); exchange.sendResponseHeaders(200, body.length); try (OutputStream os = exchange.getResponseBody()) { os.write(body); } } The method remains minimalist: no paging, no sorting, no additional data. It uses the same parameters as the ListHandler, so both endpoints behave consistently. The only difference is in the return format – a deliberate design for efficiency and clear accountability.\nAn example of a typical client request:\nGET /list/count?code=ex-\u0026amp;url=docs\n→ { \u0026ldquo;total\u0026rdquo;: 42 }\nThe client (e.g. the Vaadin UI) can use this information to calculate the maximum number of pages and dynamically enable or disable the navigation buttons. This prevents unnecessary data transfer when the user only wants to know how many entries a filter currently delivers.\nThis lean endpoint lays the foundation for high-performance paging. The next step shows how the store responds internally to these filter requests and efficiently determines the results.\nInMemoryUrlMappingStore – Filtering, sorting, and paginating # The InMemoryUrlMappingStore, which already served as a simple storage solution in [Part II], has now been extended by a powerful filter logic. The goal is to process targeted queries directly in memory using the new UrlMappingFilter, including sorting and paging. The code remains easy to understand and test, as no external database is used.\nCompared to the original variant, which provided only findAll(), the new implementation has two central extensions:\nfind(UrlMappingFilter filter) – returns a filtered, sorted, and paginated list of results. count(UrlMappingFilter filter) – determines the number of elements that correspond to a given filter. An excerpt shows the core of the new logic:\n@Override public List\u0026lt;ShortUrlMapping\u0026gt; find(UrlMappingFilter filter) { Objects.requireNonNull(filter, \u0026#34;filter\u0026#34;); List\u0026lt;ShortUrlMapping\u0026gt; tmp = new ArrayList\u0026lt;\u0026gt;(); for (ShortUrlMapping mapping : store.values()) { if (matches(filter, mapping)) tmp.add(mapping); } Comparator\u0026lt;ShortUrlMapping\u0026gt; cmp = buildComparator(filter); if (cmp != null) { tmp.sort(cmp); if (filter.direction().orElse(Direction.ASC) == Direction.DESC) { Collections.reverse(tmp); } } int from = Math.max(0, filter.offset().orElse(0)); int lim = filter.limit().orElse(Integer.MAX_VALUE); if (from \u0026gt;= tmp.size()) return List.of(); int to = Math.min(tmp.size(), from + Math.max(0, lim)); return tmp.subList(from, to); } The process is divided into three phases:\nFilter – matches(filter, mapping) checks substrings (with optional case sensitivity) as well as time periods (createdFrom, createdTo). Sort – the Comparator construct allows sorting by CREATED_AT, SHORT_CODE, ORIGINAL_URL, or EXPIRES_AT. Paging – offset and limit return only the requested subsets. An example illustrates the effect:\nFilter: codePart=\u0026ldquo;ex-\u0026rdquo;, sortBy=CREATED_AT, direction=DESC, offset=0, limit=5\nResult: The five most recent shortcodes that start with \u0026ldquo;ex-\u0026rdquo;.\nThe count(filter) method , on the other hand, is deliberately trivial:\n@Override public int count(UrlMappingFilter filter) { int c = 0; for (ShortUrlMapping m : store.values()) { if (matches(filter, m)) C++; } return c; } This keeps the semantics consistent: count() and find() use the same filter logic, but differ in the return form. This allows pagination information to be reliably calculated.\nA key design goal was backward compatibility : the original findAll() method remains unchanged. If no filter is passed, find() can fall back on it internally. Thus, older parts of the application continue to work unchanged.\nThe expansion of the store is an example of how more complex search mechanisms can be implemented using simple Java constructs – type-safe, transparent, and without additional dependencies. The following section shows how this logic is specifically supported and validated by QueryUtils.\nQueryUtils – Parsing and Normalising Query Parameters # To enable REST endpoints to respond cleanly and robustly to parameter requests, a small helper class, QueryUtils, has been introduced. It converts raw query strings from HTTP requests into type-safe values. This step relieves the handler classes (ListHandler and ListCountHandler) and at the same time ensures uniform behavior when dealing with user parameters.\nThe class is located in the package com.svenruppert.urlshortener.api.utils and is purely static. Their purpose is to catch erroneous input, set default values, and convert strings into concrete enums or numbers. This prevents incomplete or incorrect query parameters from causing exceptions or leading to inconsistent states.\nAn excerpt from the implementation:\npublic final class QueryUtils { private QueryUtils() { } public static int parseIntOrDefault(Strings s, int def) { try { return (s == null || s.isBlank()) ? def: Integer.parseInt(s.trim()); } catch (NumberFormatException e) { return def; } } public static int clamp(int v, int lo, int hi) { return Math.max(lo, Math.min(hi, v)); } public static Optional\u0026lt;UrlMappingFilter.SortBy\u0026gt; parseSort(Strings) { if (s == null) return Optional.empty(); return switch (s.toLowerCase(Locale.ROOT)) { case \u0026#34;createdat\u0026#34; -\u0026gt; Optional.of(UrlMappingFilter.SortBy.CREATED_AT); case \u0026#34;shortcode\u0026#34; -\u0026gt; Optional.of(UrlMappingFilter.SortBy.SHORT_CODE); case \u0026#34;originalurl\u0026#34; -\u0026gt; Optional.of(UrlMappingFilter.SortBy.ORIGINAL_URL); case \u0026#34;expiresat\u0026#34; -\u0026gt; Optional.of(UrlMappingFilter.SortBy.EXPIRES_AT); default -\u0026gt; Optional.empty(); }; } public static Optional\u0026lt;UrlMappingFilter.Direction\u0026gt; parseDir(Strings) { if (s == null) return Optional.empty(); return switch (s.toLowerCase(Locale.ROOT)) { case \u0026#34;asc\u0026#34; -\u0026gt; Optional.of(UrlMappingFilter.Direction.ASC); case \u0026#34;desc\u0026#34; -\u0026gt; Optional.of(UrlMappingFilter.Direction.DESC); default -\u0026gt; Optional.empty(); }; } } This small utility class does several things at once:\nResilience: Incorrect or missing parameters never lead to exceptions. Consistency: Sorting and direction are interpreted consistently throughout the system. Maintainability: Traders no longer have to worry about low-level parsing. Clamping (clamp(pageSize, 1, 500)) is a crucial security feature: it prevents oversized queries and thus protects both the server and the UI from excessive data volume.\nAn example illustrates the practical effect:\nInput: size=-5 → Result: size=1\nInput: size=9999 → Result: size=500\nThis makes QueryUtils an inconspicuous but essential part of API robustness. Its features follow the same guiding principle that runs through the entire project: explicit typing, clear boundaries, and defensive processing.\nIntroduction to API Endpoints # Now that the server-side logic has been extended to include filtering, sorting and paging, the REST API itself is now taking centre stage. It serves as the link between the internal functions of the URL shortener and the client\u0026rsquo;s or UI\u0026rsquo;s external access. This chapter aims to describe the new and extended endpoints in detail and explain their functions within the overall system.\nThe focus is not only on the data format of the answers, but also on the semantics of the supported parameters. Both aspects are crucial to make filter queries consistent, comprehensible and efficient. Thanks to a clearly structured API, both existing clients can continue to work, and new components – such as the Vaadin-based administration interface – can access subsets of data in a targeted manner without overloading the system.\nStructure of the new endpoints # With the introduction of filtering and paging, the REST API for the URL shortener has also been extended. In addition to the familiar endpoints from [Part II], two new paths are now available that are specifically designed for targeted queries:\nGET /list – returns filtered and paginated results GET /list/count – returns only the number of hits Both endpoints use the same set of query parameters. While /list returns the actual mappings in structured JSON, /list/count provides a lightweight way to determine the total set for a query. This separation deliberately follows the single-responsibilityprinciple and, at the same time, supports high-performance UIs that load pages and large amounts of data.\nA typical call to the new /list endpoint might look like this:\nGET /list?code=ex-\u0026amp;url=docs\u0026amp;page=2\u0026amp;size=25\u0026amp;sort=createdAt\u0026amp;dir=desc The result is a JSON object with the following keys:\n{ \u0026#34;mode\u0026#34;: \u0026#34;filtered\u0026#34;, \u0026#34;page\u0026#34;: 2, \u0026#34;size\u0026#34;: 25, \u0026#34;total\u0026#34;: 123, \u0026#34;sort\u0026#34;: \u0026#34;createdAt\u0026#34;, \u0026#34;dir\u0026#34;: \u0026#34;desc\u0026#34;, \u0026#34;count\u0026#34;: 25, \u0026#34;items\u0026#34;: [ { \u0026#34;shortCode\u0026#34;: \u0026#34;ex-beta\u0026#34;, \u0026#34;originalUrl\u0026#34;: \u0026#34;https://example.org/blog\u0026#34;, \u0026#34;createdAt\u0026#34;: \u0026#34;2025-10-24T12:45:33Z\u0026#34;, \u0026#34;expiresAt\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;Status\u0026#34;: \u0026#34;Active\u0026#34; } ] } The JSON structure follows a clear logic: all metadata for the request is in the first fields, while the items array contains the actual results. This makes it easy to integrate the structure into UI components such as tables, grids or data providers.\nCompatibility and stability # Existing endpoints such as /list/all, /list/active, and /list/expired are fully preserved. They continue to provide unchanged responses, so existing clients don\u0026rsquo;t need any customisations. The new endpoints are therefore integrated additively into the system.\nFor new clients, especially those using the Vaadin UI presented in [Part III], the introduction of these endpoints provides a basis for reactive data display. Instead of loading all mappings at once, the interface can request only the data sets defined by the current filter.\nThis API structure lays the foundation for detailing the supported parameters and their meanings in the following subchapters.\nSupported Parameters # The new endpoints /list and /list/count accept a set of query parameters that can be used to formulate targeted search and filter queries. All parameters are optional and can be freely combined. Unset fields do not impose any restrictions.\nParameter Type Description code String Substring of the short code. Is handled by default case-insensitive if codeCase=true is not set. codeCase Boolean Controls whether the search for shortcodes should be case-sensitive. URL String Substring within the original URL. Similar to code, the case can be controlled via urlCase. urlCase Boolean Case sensitivity switch for URL search. from ISO-8601 timestamp Lower limit of the creation period (inclusive). Also accepts dates without a time component. To ISO-8601 timestamp Upper limit of the creation period (included). page Int 1-based page number. Default value: 1. Size Int Number of records per page. Values outside the range 1-500 are automatically limited. sort String Sort. Allowed are: createdAt, shortCode, originalUrl, expiresAt. you String Sorting direction: asc or desc. Default value: asc. For example, a complete request might look like this:\nGET /list?code=ex-\u0026amp;url=docs\u0026amp;from=2025-10-01T00:00:00Z\u0026amp;to=2025-10-25T23:59:00Z\u0026amp;page=2\u0026amp;size=25\u0026amp;sort=createdAt\u0026amp;dir=desc The API validates all parameters using the QueryUtils class. Invalid or missing values are replaced with default values to avoid exceptions. This defensive strategy ensures a high level of stability in continuous operation.\nInteraction of the parameters # If neither code nor url is set, all mappings are taken into account. from and to define an inclusive period ; both fields may be set independently of each other. page and size only affect the display of results, not the count in /list/count. Combinations of sort and dir have a consistent effect on both endpoints (/list and /list/count). A minimalist example of a count-only query is:\nGET /list/count?url=example → { \u0026#34;total\u0026#34;: 12 } The API is designed so that future parameters can be easily added. Thanks to the central UrlMappingFilter, new fields need only be stored there and taken into account in the builder. This completely preserves the system\u0026rsquo;s extensibility.\nClient-side extensions # After the server-side has been extended with flexible filtering and paging functions, the client-side customisation now follows. This chapter aims to demonstrate how the new data retrieval capabilities have been integrated into the existing URLShortenerClient without altering its structure or semantics.\nThe focus is on type-safe communication between the client and the server. Instead of assembling manual query strings or passing loose maps, the new builder UrlMappingListRequest encapsulates all parameters in a clear, object-oriented form. On this basis, filters, sorting and paging can be centrally managed and easily tested.\nThe chapter highlights the three main aspects of client extension:\nThe new request builder (UrlMappingListRequest), which cleanly encapsulates filter and paging parameters. The advanced methods in the**URLShortenerClient**process these requests and forward them to the new endpoints. The associated test class that secures the interaction between client and server. Together, these elements serve as the counterpart to the server-side extensions from the chapter \u0026ldquo;Server-side changes\u0026rdquo; and provide the basis for an interactive user interface that works specifically with filtered and paginated data.\nUrlMappingListRequest – Filter and Paging Request Builder # To enable the client to communicate with the new filter and paging endpoints in a targeted manner, the UrlMappingListRequest class was introduced. It acts as a transportable request object that contains all relevant parameters and, if necessary, translates them into a query string for HTTP communication.\nThe design follows the principles from [Part II]: no external JSON serialisation, no dependencies on frameworks – instead pure Java logic with a focus on readability and type safety.\nAn example illustrates the usage:\nvar req = UrlMappingListRequest.builder() .urlPart(\u0026#34;docs\u0026#34;) .page(2) .size(25) .sort(\u0026#34;createdAt\u0026#34;) .dir(\u0026#34;desc\u0026#34;) .build(); The class then converts this information into a URL query string that can be sent directly to the server:\n/list?url=docs\u0026amp;page=2\u0026amp;size=25\u0026amp;sort=createdAt\u0026amp;dir=desc Internally, UrlMappingListRequest consists of a set of simple fields, such as codePart, urlPart, from, to, page, size, sort, and dir. The integrated builder ensures that only valid combinations can be formed. Unset values are automatically ignored during serialisation – empty parameters do not appear in the query string.\nTwo methods are central:\n**toQueryString()** – generates the complete query string including paging and sorting parameters. **toQueryStringForCount()** – returns only the filter parameters, without paging or sorting information, for /list/count. Both variants are based on an internal helper method that checks parameters, sorts them by key, and securely encodes them via URLEncoder:\nprivate static String toQuery(Map\u0026lt;String, String\u0026gt; params) { return params.entrySet().stream() .map(e -\u0026gt; enc(e.getKey()) + \u0026#34;=\u0026#34; + enc(e.getValue())) .collect(Collectors.joining(\u0026#34;\u0026amp;\u0026#34;)); } This structure ensures that all parameters are transmitted cleanly and in accordance with the URL format – a detail particularly essential for strings containing special characters (e.g., in URLs).\nThus, UrlMappingListRequest forms the direct bridge between the client and the API. It does not replace existing calls, but expands the communication options with flexible, type-safe filter queries – in line with the modular system design.\nExtension of the URLShortenerClient – Filter and Count # On the client side, the URLShortenerClient has been enhanced with two essential features to target the new server-side endpoints: list(UrlMappingListRequest request) and listCount(UrlMappingListRequest request). These methods enable dynamic reaction to filter criteria without manual URL assembly.\nNew methods # The implementation follows the principle of clear separation of responsibilities: list() handles the actual data, listCount() handles the metadata.\npublic List\u0026lt;ShortUrlMapping\u0026gt; list(UrlMappingListRequest request) throws IOException { final String json = listAsJson(request); return parseItemsAsMappings(json); } public int listCount(UrlMappingListRequest req) throws IOException { String qs = (req == null) ? \u0026#34;\u0026#34; : req.toQueryStringForCount(); var uri = qs.isEmpty() ? serverBaseAdmin.resolve(PATH_ADMIN_LIST_COUNT) : serverBaseAdmin.resolve(PATH_ADMIN_LIST_COUNT + \u0026#34;?\u0026#34; + qs); var con = (HttpURLConnection) uri.toURL().openConnection(); con.setRequestMethod(\u0026#34;GET\u0026#34;); con.setRequestProperty(\u0026#34;Accept\u0026#34;, \u0026#34;application/json\u0026#34;); int sc = con.getResponseCode(); if (sc != 200) { String err = readAllAsString(con.getErrorStream() != null ? con.getErrorStream() : con.getInputStream()); throw new IOException(\u0026#34;Unexpected HTTP \u0026#34; + sc + \u0026#34; for \u0026#34; + uri + \u0026#34; body=\u0026#34; + err); } try (var is = con.getInputStream()) { String body = readAllAsString(is); int i = body.indexOf(\u0026#34;\\\u0026#34;total\\\u0026#34;\u0026#34;); if (i \u0026lt; 0) return 0; int colon = body.indexOf(\u0026#39;:\u0026#39;, i); int end = body.indexOf(\u0026#39;}\u0026#39;, colon + 1); String num = body.substring(colon + 1, end).trim(); return Integer.parseInt(num); } finally { con.disconnect(); } } Significance of enlargement # These two methods make the client fully compatible with the advanced server features. The filter logic is completely mapped via the UrlMappingListRequest object, so that the call itself always remains clear:\nvar req = UrlMappingListRequest.builder() .codePart(\u0026#34;ex-\u0026#34;) .urlPart(\u0026#34;docs\u0026#34;) .page(1) .size(25) .build(); List\u0026lt;ShortUrlMapping\u0026gt; results = client.list(req); int total = client.listCount(req); The client can therefore first query the total amount to calculate pagination, then load only the relevant data pages. This significantly reduces the network load, especially with large amounts of data.\nBackward compatibility # All existing methods such as listAllAsJson() or listActiveAsJson() are preserved. This keeps the API binary and semantically compatible – existing tests and applications continue to work. The new functions fit seamlessly into the existing design and, at the same time, form the basis for a reactive UI that can dynamically evaluate filters and paging.\nClient Test Coverage – URLShortenerClientListTest # To secure the new client functions, a separate test class, URLShortenerClientListTest, has been introduced. It checks the interaction between the client and a running ShortenerServer and thus serves as an integration test for the entire filter and paging chain.\nThe test starts a whole server on a random port and then communicates with the real HTTP endpoint. This ensures that not only the client logic, but also the serialisation, request routing, and server-side filtering work correctly.\nAn excerpt from the test case:\n@Test @Order(1) void list_all_and_filtered_by_code_and_url_and_date() throws Exception { Arrange var t0 = Instant.now(); ShortUrlMapping m1 = client.createCustomMapping(\u0026#34;ex-alpha\u0026#34;, \u0026#34;https://example.com/docs\u0026#34;); Thread.sleep(10); ShortUrlMapping m2 = client.createCustomMapping(\u0026#34;ex-beta\u0026#34;, \u0026#34;https://example.org/blog\u0026#34;); Thread.sleep(10); ShortUrlMapping m3 = client.createMapping(\u0026#34;https://docs.example.com/page\u0026#34;); var t1 = Instant.now(); Act – test different filters var reqCode = UrlMappingListRequest.builder().codePart(\u0026#34;ex-\u0026#34;).build(); List\u0026lt;ShortUrlMapping\u0026gt; byCode = client.list(reqCode); assertTrue(byCode.size() \u0026gt;= 2); var reqUrl = UrlMappingListRequest.builder().urlPart(\u0026#34;docs\u0026#34;).build(); List\u0026lt;ShortUrlMapping\u0026gt; byUrl = client.list(reqUrl); assertTrue(byUrl.stream().anyMatch(m -\u0026gt; m.originalUrl().contains(\u0026#34;docs\u0026#34;))); var reqDate = UrlMappingListRequest.builder().from(t0).to(t1).build(); List\u0026lt;ShortUrlMapping\u0026gt; byDate = client.list(reqDate); assertFalse(byDate.isEmpty()); } The test examines three central use cases:\nCode filtering – only shortcodes that start with a specific pattern. URL filtering – URLs that contain specific substrings. Time Filtering – Entries created within a specific time window. Paging and sorting tests # A second test section focuses on pagination and sorting:\n@Test @Order(2) void list_pagination_and_sorting() throws Exception { for (int i = 0; i \u0026lt; 10; i++) { client.createMapping(\u0026#34;https://example.net/page-\u0026#34; + i); Thread.sleep(2); } var req = UrlMappingListRequest.builder() .page(2).size(5) .sort(\u0026#34;createdAt\u0026#34;).dir(\u0026#34;desc\u0026#34;) .build(); List\u0026lt;ShortUrlMapping\u0026gt; page2 = client.list(req); assertEquals(5, page2.size()); for (int i = 1; i \u0026lt; page2.size(); i++) { assertTrue(!page2.get(i).createdAt().isAfter(page2.get(i - 1).createdAt())); } } This checks that page numbering is working correctly and that the results are returned in the expected order.\nAim of the tests # This test class ensures that the interaction between client and server remains stable, even if parameter combinations change. By using a real HttpServer backend, a realistic environment is simulated, without external dependencies. The result is a high level of trust in the implementation with full coverage of all critical paths.\nThe URLShortenerClientListTest class thus concludes the client chapter: It proves that the system as a whole – from filter definition to HTTP communication to result processing – functions consistently.\nIn the next part, we will then look at the integration into the Vaadin UI.\nCheers Sven\n","date":"2 December 2025","externalUrl":null,"permalink":"/posts/advent-calendar-2025-filter-search-part-01/","section":"Posts","summary":"With the Vaadin interface described in [Part III], our URL shortener has a fully functional administration console available for the first time. It allows viewing existing short links in tabular form and managing them manually. But after just a few dozen entries, a clear limit becomes apparent: displaying all saved mappings is neither performant nor user-friendly. An efficient shortener must be able to scale – not only when generating, but also when searching through its data.\n","title":"Advent Calendar - 2025 - Filter \u0026amp; Search – Part 01","type":"posts"},{"content":"","date":"2 December 2025","externalUrl":null,"permalink":"/categories/design-pattern/","section":"Categories","summary":"","title":"Design Pattern","type":"categories"},{"content":"December 2025 is all about a project that has grown steadily in recent months: the Java-based URL Shortener, an open-source project implemented entirely with Core Java, Jetty, and Vaadin Flow. The Advent calendar accompanies users every day with a new feature, a technical deep dive, or an architectural improvement – from the basic data structure and REST handlers to UI components and security aspects.\nMotivation: Why your own URL shortener? What can users expect in the Advent calendar? Reasons for the choice of technology Core Java (Java 24) Jetty as an embedded web server Vaadin Flow for the UI EclipseStore as a persistence solution Reference to previously published articles Why, as an Advent calendar? Motivation: Why your own URL shortener? # The entire project is developed openly on GitHub. There, users will not only find the current state of the code, but also all historical development steps, patches, branches, and PRs discussed in the context of the Advent calendar. The source code can be viewed at https://github.com/svenruppert/url-shortener.\nCommercial URL shorteners are practical – but often non-transparent. They collect usage data, offer proprietary APIs, or block essential functions behind paywalls. The goal of this project is therefore a completely self-powered, transparent, and secure shortener suitable for internal networks, development teams, companies, and hobby projects alike.\nThe project pursues three central basic principles: transparency, in which every decision is documented in the code in a comprehensible way; didactics, as the project is not only intended to be a tool, but above all a learning platform; and extensibility, because each feature is designed to be modular and can therefore be easily adapted or expanded.\nWhat can users expect in the Advent calendar? # The technical journey through Advent is visible both in terms of content and directly in the repository. Each day is created as its own Git branch, starting with feature/advent-2025-day-01 and continuing until the final Christmas door. This allows the project\u0026rsquo;s development to be traced in small, clearly defined steps – including all refactorings, architectural adjustments, and new functions. In the same structure, the Advent Calendar presents a precisely outlined feature every day, which is reproduced using the associated patch or PR. Supplementary code excerpts from the core and UI modules, architectural explanations and references to typical anti-patterns deepen the understanding. Each door is rounded off with suitable visualisations and a compact summary, so that the project\u0026rsquo;s development becomes transparent and tangible step by step.\nReasons for the choice of technology # For the URL shortener, technologies were deliberately chosen that are both practical and didactically valuable. Every decision supports the project\u0026rsquo;s goal: to create a modern yet easy-to-understand system that requires no unnecessary dependencies.\nCore Java (Java 24) # The project\u0026rsquo;s basis is pure Core Java. The reasons for this are:\nThe use of pure core Java enables maximum transparency: Without any framework magic, users can see exactly how data models, handlers, error handling or serialisation are structured. By deliberately avoiding additional libraries, the attack surface is reduced, and the risk of supply chain problems is minimised. At the same time, this approach makes it very clear to developers how robust and comprehensive the JDK already is today: Many tasks that used to require external dependencies can now be easily solved with onboard tools.\nJetty as an embedded web server # A deliberate alternative to Spring Boot or Jakarta EE:\nJetty is characterised by its lightweight design and rapid launch time, which are especially beneficial in development scenarios. At the same time, Jetty offers complete control over routing and servlets, so that HTTP mechanics can be precisely demonstrated and implemented in a targeted manner. Thanks to its modular structure, Jetty is ideal for small, well-defined microservices while remaining a stable, proven technology whose long production history speaks for high reliability.\nJetty offers just the right balance between simplicity and technical relevance for this open source project.\nVaadin Flow for the UI # The project uses Vaadin Flow to implement an entirely server-side, Java-focused UI that does not require an additional JavaScript or TypeScript stack, making it particularly suitable for developers who want to focus entirely on Java. Instead of relying on a complex front-end ecosystem, Vaadin enables end-to-end development in a language, significantly reducing the cognitive load and flattening the learning curve. The component-based architecture enables a productive and structured way of working, in which user interfaces can be clearly modelled and reused. At the same time, server-side rendering eliminates the need for direct REST calls from the browser, increasing security and reducing attack surfaces. Despite this technical simplicity, Vaadin offers modern styling and a user experience that is reminiscent of the professionalism of enterprise applications. This makes the framework ideal for internal tools, administrative interfaces and corporate projects where security, robustness and long-term maintainability are paramount.\nEclipseStore as a persistence solution # EclipseStore replaces classic databases with an object-oriented persistence approach that deliberately does not require ORM layers, entities, table models or complex abstractions. Instead of having to convert data into a relational structure, it remains in its natural form: Java records, lists and other object structures are directly persisted. This eliminates the usual mapping logic, significantly reducing the complexity of the persistence layer and making the architecture more straightforward to understand.\nEclipseStore shows its strength especially in small, focused services. Persistence fits seamlessly into the domain model without forcing developers to adapt their data structures to a relational mindset. This direct approach not only leads to more elegant modelling but also to excellent performance. Since EclipseStore manages in-memory-compliant object graphs, data access is high-speed and does not require any additional caching mechanisms or complex optimisations. The result is a lightweight yet powerful persistence system ideal for a modern, compact Java service.\nDidactically, EclipseStore provides valuable insight into how persistent data models can operate when not constrained by relational database constraints. Developers gain a deeper understanding of what object lifecycles, data flows, and modelling decisions look like in an environment that is designed entirely from the Java world. This makes EclipseStore particularly suitable for this URL shortener.\nReference to previously published articles # Before we get into the Advent calendar, it\u0026rsquo;s worth taking a look at the previous publications, which already present the basics, architecture and the first UI concepts of the project. These articles form the foundation on which the content of the Advent calendar is built:\nPart I – Short links, clear architecture : A detailed introduction to the motivation, architecture and data model of the project. The article shows how to develop a URL shortener entirely in Core Java, along with the design principles behind it. (https://svenruppert.com/2025/06/10/short-links-clear-architecture-a-url-shortener-in-core-java/ ) Part II – Deepening the Server Components : This article explores REST handlers, validation, security considerations, and the interaction between the core module and the server layer. (https://svenruppert.com/2025/06/20/part-ii-urlshortener-first-implementation/) Part III – The Web UI with Vaadin Flow: This shows how the user interface is structured, how lists, forms, dialogues and validations work, and how Vaadin Flow enables a modern UI without an additional JavaScript stack. (https://svenruppert.com/2025/08/15/part-iii-webui-with-vaadin-flow-for-the-url-shortener/) These items offer a first technical introduction and are perfect as preparation for the individual doors of the Advent calendar.\nWhy, as an Advent calendar? # A project like a URL shortener doesn\u0026rsquo;t happen in a weekend – it consists of many small steps, refactorings, decisions and improvements. An Advent calendar offers the perfect format for this: short, daily, understandable bites that come together to form an overall understanding.\nIt is, at the same time, a review of the project\u0026rsquo;s journey and a motivating start to the new year: a project that users can run, expand, change, or completely rethink.\n","date":"1 December 2025","externalUrl":null,"permalink":"/posts/introduction-to-the-url%e2%80%91shortener-advent-calendar-2025/","section":"Posts","summary":"December 2025 is all about a project that has grown steadily in recent months: the Java-based URL Shortener, an open-source project implemented entirely with Core Java, Jetty, and Vaadin Flow. The Advent calendar accompanies users every day with a new feature, a technical deep dive, or an architectural improvement – from the basic data structure and REST handlers to UI components and security aspects.\n","title":"Introduction to the URL‑Shortener Advent Calendar 2025","type":"posts"},{"content":"","date":"1 December 2025","externalUrl":null,"permalink":"/tags/jetty/","section":"Tags","summary":"","title":"Jetty","type":"tags"},{"content":"","date":"5 September 2025","externalUrl":null,"permalink":"/tags/event-sourcing/","section":"Tags","summary":"","title":"Event Sourcing","type":"tags"},{"content":" Chapter 1 – Introduction # 1.1 Motivation: Real-time communication without polling # In modern applications, it is often necessary to provide new information to the client as quickly as possible. Classic polling , i.e. regularly querying a REST endpoint, is inefficient: it generates unnecessary network traffic and puts a load on both server and client, as requests continue even when there is no new data.\nServer-sent events (SSE) offer a resource-saving alternative here. Instead of the client constantly sending new requests, it maintains an open HTTP connection over which the server can send messages at any time. This creates a continuous stream of data with minimal overhead.\nChapter 1 – Introduction 1.1 Motivation: Real-time communication without polling 1.2 Differentiation from WebSockets and Long Polling 1.3 Objective of the article Chapter 2 – The SSE Protocol 2.1 MIME type text/event-stream 2.2 Message structure: event:, data:, id: and comments 2.3 Automatic Reconnect and Event IDs 2.4 Limitations: UTF-8, Unidirectionality, Proxy Behaviour Chapter 3 – Minimal REST/SSE Server in Core Java 3.1 Implementation with HttpServer 3.2 Sending Events (Text, Comments, IDs) 3.3 Keep-Alive and Connection Stability 3.4 Step-by-step implementation – complete minimal example Chapter 4 – Minimal SSE Client in Core Java 4.1 Implementation with HttpClient 4.2 Interpreting Events 4.3 Behaviour in case of aborting and reconnect Chapter 5 – Testing and Debugging SSE 5.1 Using curl and browser EventSource 5.2 Checking with Java Client 5.3 Monitoring and logging Chapter 6 – Application Scenarios and Limitations 1.2 Differentiation from WebSockets and Long Polling # In the context of current communication paradigms, three basic approaches can be distinguished. In so-called long polling , the client initiates a request that the server keeps open as long as possible until new information is available. After this has been transmitted, the connection is closed, whereupon the client immediately opens a new request. Although this method appears to be more efficient than classic polling, it remains resource-intensive due to the frequent reinitialization of the connections.\nIn contrast, WebSockets establish a full-duplex communication relationship. Here, both client and server can exchange messages continuously and bidirectionally. This model offers high flexibility and performance, but it also introduces increased complexity in implementation and operation, often resulting in oversizing when only server-side push messages are needed.\nServer-sent events (SSE) act as an intermediary approach. They establish a standardised, unidirectional data stream from the server to the client via a persistent HTTP connection. The native support of current browsers and the comparatively low implementation effort on the server side make SSE a practical option, especially in scenarios where simplicity and resource conservation are paramount.\nSourcecode is on GitHub\nhttps://github.com/Java-Publications/Blog---Java---Server-Sent-Events-SSE-in-Core-Java-Basics-and-Implementation\n1.3 Objective of the article # This article provides a basic introduction to Server-Sent Events (SSE) using Core Java. The focus is on consistent implementation based on the Java Development Kit (JDK), without recourse to external frameworks. The aim is to analyse the specifics of the protocol as well as the minimal implementations on both the server side and the client side in a systematic form and to present them in a comprehensible manner.\nIn the course of the study, it is illustrated how an elementary REST/SSE server can be constructed in Java, how a client can receive and process event streams using the Java HttpClient, which typical difficulties arise in the context of testing and debugging, and in which application scenarios the use of SSE offers significant added value.\nFinally, the position of SSE in the area of tension between polling and WebSocket-based methods is critically classified and the transition to the following, practice-oriented implementation chapters is prepared.\nChapter 2 – The SSE Protocol # 2.1 MIME type text/event-stream # The foundation of server-sent events is the dedicated MIME type text/event-stream. As soon as a server declares this Content-Type in the HTTP header, it signals to the client that the subsequent data is to be interpreted as a continuous event stream. In contrast to conventional textual response formats, which are closed after complete transmission, SSE deliberately keeps the connection open persistently. This semantic definition creates the basis for a resource-saving push model that can be integrated into existing HTTP infrastructures without proprietary extensions.\n2.2 Message structure:event: , data: , id: and comments # An SSE data stream is organised on a row-based basis. Each message can consist of multiple fields preceded by a keyword and a colon. The most important fields are:\nevent: defines the type of event that the client can handle specifically. data: contains the actual payload. Consecutive data lines summarise multi-line data fields. id: represents an event ID that is used for resumption after disconnections. Comments: start with a colon and are used both for semantic documentation and to maintain keep-alive signals. The delimitation of individual events is done by a blank line (double line breaks), which completes the semantic unit of a message.\n2.3 Automatic Reconnect and Event IDs # A significant advantage of SSE over simpler push mechanisms is the integrated reconnect behaviour. If a connection is unexpectedly interrupted, the client automatically initiates a new connection. Using the Last Event ID HTTP header fields or explicit ID assignments in the data stream, it is possible to continue at the exact point where the data stream was interrupted. This principle reduces data loss and facilitates the implementation of robust, state-preserving communication patterns.\n2.4 Limitations: UTF-8, Unidirectionality, Proxy Behaviour # Despite its simplicity, the protocol has inherent limitations. On the one hand, the character encoding is strictly set to UTF-8, which means that binary payloads cannot be transmitted directly. In addition, the direction of communication is limited to server-side messages; a bidirectional interaction requires recourse to alternative methods such as WebSockets. Finally, the behaviour of intermediary network nodes – such as proxies or load balancers – can affect the longevity and stability of open HTTP connections, which deserves special attention in production environments.\nChapter 3 – Minimal REST/SSE Server in Core Java # 3.1 Implementation with HttpServer # An elementary SSE server can be implemented in Java using the classes included in the JDK. Of particular note is the com.sun.net.httpserver.HttpServer class, which allows HTTP endpoints to be provided without external dependencies. Such a server can be created with a few lines of code, bound to a port number and extended by handlers that process incoming requests.\nA minimal basic framework looks like this:\nHttpServer server = HttpServer.create(new InetSocketAddress(8080), 0); server.createContext(\u0026#34;/events\u0026#34;, exchange -\u0026gt; { exchange.getResponseHeaders().add(\u0026#34;Content-Type\u0026#34;, \u0026#34;text/event-stream\u0026#34;); exchange.sendResponseHeaders(200, 0); OutputStream os = exchange.getResponseBody(); //This is where the events will be written later }); server.start(); It is crucial to set the content type to text/event-stream and not to close the HTTP connection immediately. This creates the basis for continuously transmitting event data to the client.\n3.2 Sending Events (Text, Comments, IDs) # The server-side logic for transmitting messages follows the formats defined in the protocol. A typical message can contain both a data: field with payload and optional metadata. Comments are beneficial for realising keep-alive signals.\nA simple example of how to transmit messages:\nPrintWriter writer = new PrintWriter(os, true); writer.println(\u0026#34;id: 1\u0026#34;); writer.println(\u0026#34;event: greeting\u0026#34;); writer.println(\u0026#34;data: Hello client!\u0026#34;); writer.println(); End message writer.flush(); //Comment as a heartbeat writer.println(\u0026#34;: keep-alive\u0026#34;); writer.println(); writer.flush(); Here, a message is transmitted with ID, event type, and data. The double line break concludes the semantic unit. The comment is to maintain the connection.\n3.3 Keep-Alive and Connection Stability # A critical aspect of running SSE servers is ensuring connection stability. HTTP connections can be terminated by inactivity or by restrictive network infrastructures. To avoid this, it is common practice to periodically send comments or blank messages that are ignored by the client but registered as activity by the network. This technique, often referred to as heartbeat or keep-alive, helps keep the connection stable for extended periods of time.\nA robust implementation should also ensure that there are no resource leaks when the connection is lost. In particular, this means that OutputStreams must be reliably closed and background threads must be terminated in a controlled manner.\nThis step-by-step implementation creates a functional SSE server that meets the fundamental protocol requirements. In the following chapters, this is built on by developing a corresponding client and examining the robustness of the communication.\n3.4 Step-by-step implementation – complete minimal example # In the following, an executable minimal server is set up in clearly defined steps. The implementation uses only the JDK (no external dependencies) and relies on com.sun.net.httpserver.HttpServer.\n**Create and start the server\npackage com.svenruppert.sse; import com.sun.net.httpserver.HttpServer; import com.svenruppert.dependencies.core.logger.HasLogger; import java.io.*; import java.net.InetSocketAddress; import static java.nio.charset.StandardCharsets.UTF_8; public class SseServer implements HasLogger { protected static final String PATH_SSE = \u0026#34;/sse\u0026#34;; private volatile boolean sendMessages = false; private volatile boolean shutdownMessage = false; // Reference so we can properly stop the server from the CLI private HttpServer server; public void start() throws Exception { int port = 8080; this.server = HttpServer.create(new InetSocketAddress(port), 0); // Background thread for CLI control (start | stop | shutdown) Thread cli = new Thread(() -\u0026gt; { try (BufferedReader console = new BufferedReader(new InputStreamReader(System.in))) { String line; while ((line = console.readLine()) != null) { String cmd = line.trim().toLowerCase(); switch (cmd) { case \u0026#34;start\u0026#34; -\u0026gt; cmdStart(); case \u0026#34;stop\u0026#34; -\u0026gt; cmdStop(); case \u0026#34;shutdown\u0026#34; -\u0026gt; cmdShutdown(); default -\u0026gt; logger().info(\u0026#34;Unknown command: {} (allowed: start | stop | shutdown)\u0026#34;, cmd); } } } catch (IOException ioe) { logger().warn(\u0026#34;CLI control terminated: {}\u0026#34;, ioe.getMessage()); } }, \u0026#34;cli-control\u0026#34;); cli.setDaemon(true); cli.start(); server.createContext(PATH_SSE, exchange -\u0026gt; { if (!\u0026#34;GET\u0026#34;.equalsIgnoreCase(exchange.getRequestMethod())) { exchange.sendResponseHeaders(405, -1); return; } exchange.getResponseHeaders().add(\u0026#34;Content-Type\u0026#34;, \u0026#34;text/event-stream; charset=utf-8\u0026#34;); exchange.getResponseHeaders().add(\u0026#34;Cache-Control\u0026#34;, \u0026#34;no-cache\u0026#34;); exchange.getResponseHeaders().add(\u0026#34;Connection\u0026#34;, \u0026#34;keep-alive\u0026#34;); exchange.sendResponseHeaders(200, 0); try (OutputStream os = exchange.getResponseBody(); OutputStreamWriter osw = new OutputStreamWriter(os, UTF_8); BufferedWriter bw = new BufferedWriter(osw); PrintWriter out = new PrintWriter(bw, true)) { long id = 0; while (!shutdownMessage) { if (sendMessages) { id++; String data = \u0026#34;tick @\u0026#34; + System.currentTimeMillis(); writeEvent(out, \u0026#34;tick\u0026#34;, data, Long.toString(id)); } else { heartbeat(out); } Thread.sleep(1000); } // Optional: send farewell event before shutting down writeEvent(out, \u0026#34;shutdown\u0026#34;, \u0026#34;Server is shutting down\u0026#34;, Long.toString(++id)); } catch (IOException | InterruptedException ioe) { logger().warn(\u0026#34;SSE client disconnected: {}\u0026#34;, ioe.getMessage()); } finally { logger().info(\u0026#34;SSE stream closed\u0026#34;); } }); server.start(); logger().info(\u0026#34;SSE server running at http://localhost:{}/sse\u0026#34;, port); Runtime.getRuntime().addShutdownHook(new Thread(() -\u0026gt; { logger().info(\u0026#34;Stopping server …\u0026#34;); if (server != null) { server.stop(0); } })); } private void cmdShutdown() { logger().info(\u0026#34;Shutdown command received – \u0026#39;shutdown\u0026#39; event will be sent …\u0026#34;); shutdownMessage = true; // signals all active handlers to send a shutdown event try { // Grace period so handlers can still execute writeEvent(..., \u0026#34;shutdown\u0026#34;, ...) + flush Thread.sleep(1200); } catch (InterruptedException ie) { Thread.currentThread().interrupt(); } try { if (server != null) { server.stop(0); // stop HTTP server afterwards } } catch (Exception e) { logger().warn(\u0026#34;Error while stopping: {}\u0026#34;, e.getMessage()); } logger().info(\u0026#34;Application is shutting down.\u0026#34;); System.exit(0); } private void cmdStop() { sendMessages = false; logger().info(\u0026#34;SSE message sending stopped via CLI\u0026#34;); } private void cmdStart() { sendMessages = true; logger().info(\u0026#34;SSE message sending started via CLI\u0026#34;); } // Helper function for sending an event in SSE format private void writeEvent(PrintWriter out, String event, String data, String id) { if (event != null \u0026amp;\u0026amp; !event.isEmpty()) { out.printf(\u0026#34;event: %s%n\u0026#34;, event); } if (id != null \u0026amp;\u0026amp; !id.isEmpty()) { out.printf(\u0026#34;id: %s%n\u0026#34;, id); } if (data != null) { // Correctly output multiline data for (String line : data.split(\u0026#34;\\\\R\u0026#34;, -1)) { out.printf(\u0026#34;data: %s%n\u0026#34;, line); } } out.print(\u0026#34;\\n\u0026#34;); // terminate message with empty line out.flush(); } // Comment-based heartbeat, ignored by client – keeps connection alive private void heartbeat(PrintWriter out) { out.print(\u0026#34;: keep-alive\\n\\n\u0026#34;); out.flush(); } } **Quick function test\nStart: java Application (or compile/execute via javac/java). Check with a small Java program: import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import static java.net.http.HttpClient.newHttpClient; public class SseTestClient { public static void main(String[] args) throws Exception { // Create a new HTTP/2 client HttpClient client = newHttpClient(); // Build GET request to connect to the SSE endpoint HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(\u0026#34;http://localhost:8080/sse\u0026#34;)) .GET( .build(); // Asynchronously send request and consume the response stream line by line client.sendAsync(request, HttpResponse.BodyHandlers.ofLines()) .thenAccept(response -\u0026gt; response.body().forEach(System.out::println)) .join(); } } This simple program opens a connection to the server and outputs the received SSE messages line by line to the console. Continuous event:/id:/data: blocks and : keep-alive comments are expected.\nChapter 4 – Minimal SSE Client in Core Java # 4.1 Implementation with HttpClient # On the client side, the JDK has offered a powerful API since Java 11 with java.net.http.HttpClient to process HTTP connections asynchronously. Because SSE is based on a persistent GET request, the client can continuously receive incoming data via HttpClient. In contrast to classic REST requests, which are completed after the body has been entirely accepted, the connection remains open and delivers lines in text/event-stream format.\nWe encapsulate the implementation in a dedicated class SseClient, which contains no static methods. The client is then started via a separate class SseClientApp.\npackage com.svenruppert.sse.client; import com.svenruppert.dependencies.core.logger.HasLogger; import java.io.IOException; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.time.Duration; import java.util.function.Consumer; import java.util.stream.Stream; /** * Minimal, instance-based SSE client built on java.net.http.HttpClient. * - manages Last-Event-ID * - implements a reconnect loop * - parses text/event-stream */ public final class SseClient implements HasLogger, AutoCloseable { private final HttpClient http; private final URI uri; private final Duration reconnectDelay = Duration.ofSeconds(2); private volatile boolean running = true; private volatile String lastEventId = null; public SseClient(URI uri) { this.uri = uri; this.http = HttpClient.newBuilder() .connectTimeout(Duration.ofSeconds(5)) .build(); } /** * Starts the streaming loop and invokes the callback for each complete event. */ public void run(Consumer\u0026lt;SseEvent\u0026gt; onEvent) { while (running) { try { HttpRequest.Builder b = HttpRequest.newBuilder() .uri(uri) .timeout(Duration.ofMinutes(10)) .GET(); if (lastEventId != null) { b.setHeader(\u0026#34;Last-Event-ID\u0026#34;, lastEventId); } HttpRequest req = b.build(); // synchronous streaming so we can parse line by line HttpResponse\u0026lt;Stream\u0026lt;String\u0026gt;\u0026gt; resp = http.send(req, HttpResponse.BodyHandlers.ofLines()); if (resp.statusCode() != 200) { logger().warn(\u0026#34;Unexpected status {} from {}\u0026#34;, resp.statusCode(), uri); sleep(reconnectDelay); continue; } try { parseStream(resp.body(), onEvent); } catch (java.io.UncheckedIOException uioe) { // Common case on server shutdown: stream is closed → reconnect gracefully var cause = uioe.getCause(); logger().info(\u0026#34;Stream closed ({}). Reconnecting …\u0026#34;, cause != null ? cause.getClass().getSimpleName() : uioe.getClass().getSimpleName()); } catch (RuntimeException re) { logger().warn(\u0026#34;Unexpected runtime exception in parser: {}\u0026#34;, re.getMessage()); } } catch (IOException | InterruptedException e) { if (!running) break; // terminated normally logger().warn(\u0026#34;Connection interrupted ({}). Reconnecting in {}s …\u0026#34;, e.getClass().getSimpleName(), reconnectDelay.toSeconds()); sleep(reconnectDelay); } } } private void parseStream(Stream\u0026lt;String\u0026gt; lines, Consumer\u0026lt;SseEvent\u0026gt; onEvent) { String event = null, id = null; StringBuilder data = null; try { for (String line : (Iterable\u0026lt;String\u0026gt;) lines::iterator) { if (line.isEmpty()) { // finish current message if (data != null) { String payload = data.toString(); SseEvent ev = new SseEvent(event, payload, id); if (id != null) lastEventId = id; // remember progress // shutdown signal from server → client terminates gracefully if (\u0026#34;shutdown\u0026#34;.equalsIgnoreCase(ev.event())) { try { onEvent.accept(ev); } catch (RuntimeException ex) { logger().warn(\u0026#34;Event callback threw exception: {}\u0026#34;, ex.getMessage()); } this.running = false; break; // leave parsing loop } try { onEvent.accept(ev); } catch (RuntimeException ex) { logger().warn(\u0026#34;Event callback threw exception: {}\u0026#34;, ex.getMessage()); } } event = null; id = null; data = null; continue; } if (line.startsWith(\u0026#34;:\u0026#34;)) { // comment/heartbeat — silently ignore (avoid log spam) continue; } int idx = line.indexOf(\u0026#39;:\u0026#39;); String field = (idx \u0026gt;= 0 ? line.substring(0, idx) : line).trim(); String value = (idx \u0026gt;= 0 ? line.substring(idx + 1) : \u0026#34;\u0026#34;); if (value.startsWith(\u0026#34; \u0026#34;)) value = value.substring(1); // drop optional leading space switch (field) { case \u0026#34;event\u0026#34; -\u0026gt; event = value; case \u0026#34;id\u0026#34; -\u0026gt; id = value; case \u0026#34;data\u0026#34; -\u0026gt; { if (data == null) data = new StringBuilder(); if (!data.isEmpty()) data.append(\u0026#39; \u0026#39;); data.append(value); } default -\u0026gt; { /* unknown field ignored */ } } } } catch (java.io.UncheckedIOException uioe) { // Typically EOF/closed when server stops → do not throw further var cause = uioe.getCause(); logger().info(\u0026#34;Input stream ended ({}).\u0026#34;, cause != null ? cause.getClass().getSimpleName() : uioe.getClass().getSimpleName()); } } private void sleep(Duration d) { try { Thread.sleep(d.toMillis()); } catch (InterruptedException ignored) { Thread.currentThread().interrupt(); } } @Override public void close() { running = false; } } package com.svenruppert.sse.client; import com.svenruppert.dependencies.core.logger.HasLogger; import java.net.URI; public final class SseClientApp implements HasLogger { public static void main(String[] args) { String url = (args.length \u0026gt; 0 ? args[0] : \u0026#34;http://localhost:8080/sse\u0026#34;); URI uri = URI.create(url); try (SseClient client = new SseClient(uri)) { Runtime.getRuntime().addShutdownHook(new Thread(client::close)); client.run(ev -\u0026gt; { // You could branch based on ev.event(); for now we just log if (ev.id() != null) { System.out.printf(\u0026#34;[%s] id=%s data=%s%n\u0026#34;, ev.event(), ev.id(), ev.data()); } else { System.out.printf(\u0026#34;[%s] data=%s%n\u0026#34;, ev.event(), ev.data()); } }); } } } 4.2 Interpreting Events # The SseClient has a parseEvents(Stream) method that parses incoming rows for event:, data:, and id: fields and returns them as SseEvent objects. This allows the received data to be structured and processed in a targeted manner.\n4.3 Behaviour in case of aborting and reconnect # The infinite loop mechanism implemented in the start() method ensures that the client automatically establishes a new connection in the event of an abort. A short break is taken between reconnects to avoid endless loops in the event of permanent connection problems.\nWith this architecture, the client is clearly structured: SseClient encapsulates the logic, while SseClientApp is the starting point for execution. This means that the implementation remains modular, testable and platform-independent.\nHints\nSseClient has no static methods; it is controlled by instance state (running, lastEventId). SseClientApp is the entry point. The URL can be passed as an argument; Standard is http://localhost:8080/sse. Reconnect takes place automatically with the last received Last Event ID. The callback interface (Consumer) enables flexible further processing (logging, UI updates, persistence, etc.). Chapter 5 – Testing and Debugging SSE # 5.1 Using curl and browser EventSource # To check SSE endpoints, developers often turn to curl because it\u0026rsquo;s available on almost all platforms. With the -N option (no buffering) the messages can be made directly visible:\ncurl -N http://localhost:8080/sse\nThe received events appear in the terminal in their raw form, including event:, data: and id: fields. An alternative is to use the JavaScript API EventSource in the browser:\nconst source = new EventSource(\u0026ldquo;http://localhost:8080/sse\u0026rdquo;);\nsource.onmessage = e = \u0026gt; console.log(e.data);\nThis allows the behavior of the server to be tracked directly in the browser.\n5.2 Checking with Java Client # Since this article deliberately emphasizes platform independence, a separate Java client is a good idea instead of external tools (see chapter 4). This can be used both as a permanent listener and specifically for test cases, for example by consuming only a certain number of events and then terminating them.\nExample: A test run that collects exactly five events and then ends:\nSseClient client = new SseClient(URI.create(\u0026#34;http://localhost:8080/sse\u0026#34;)); List\u0026lt;SseEvent\u0026gt; events = new ArrayList\u0026lt;\u0026gt;(); client.run(ev -\u0026gt; { events.add(ev); if (events.size() \u0026gt;= 5) { client.close(); } }); This allows functional and integration tests to be carried out directly in the Java environment without dependence on curl or telnet .\n5.3 Monitoring and logging # Precise monitoring of SSE connections is indispensable for the operation of productive systems. Monitoring is the systematic recording of key figures such as the number of active clients or the average duration of a connection. Equally important is end-to-end error tracking, which documents occurring IO errors, timeouts and aborted transmissions and thus forms the basis for rapid troubleshooting. In addition, continuous monitoring of the heartbeats ensures that comments or keep-alive signals are sent at regular intervals, keeping the connections stable.\nEffective logging supports this monitoring and should be done on both the server and client sides. This makes it possible to isolate network problems or interference caused by proxies more quickly. It is beneficial to record unique identifiers for connections and individual events in the log to make communication paths transparent and comprehensible even in complex scenarios.\nThese tools can be used to reliably track, reproduce and monitor the behaviour of SSE implementations in production environments.\nChapter 6 – Application Scenarios and Limitations # Server-sent events (SSE) find their strength in fields of application where continuous, unidirectional data transmission from the server to the client is required. It is particularly obvious to use it in the field of notification systems. Here, events such as the arrival of new messages, status changes in a workflow or system warnings can be transmitted directly to users without them having to make repeated requests actively. Another typical scenario is monitoring solutions, in which measured values or system metrics are visualised in real time. SSE also offers a resource-saving option for continuous data delivery in telemetry, for example, in the transmission of sensor data from distributed systems.\nAt the same time, it is essential to consider the limitations of the procedure. Since SSE only transmits UTF-8 encoded text data, it is not possible to transmit binary content directly. Although binary data can be encoded in text form (e.g. using Base64), this is at the expense of efficiency. SSE is also limited to one-way communication: the server can send data to the client, but not vice versa. For scenarios that require bidirectional exchange, WebSockets or similar technologies are more suitable. Network Restrictive problem infrastructures represent another problem area. Proxies or firewalls can block or disconnect long-term open HTTP connections after a certain amount of time, necessitating an additional reconnect strategy.\nSafety aspects should also not be neglected when using SSE. Since the connection is maintained via the HTTP protocol, particular attention must be paid to transport encryption using HTTPS. In addition, mechanisms should be implemented to prevent misuse by unauthorised clients, such as authentication and access control to the SSE endpoint. After all, resource conservation is a central issue: While SSE works much more efficiently than polling, the simultaneous supply of a massive number of clients can lead to a high load on server resources. Scaling strategies such as load balancing or splitting clients across multiple SSE endpoints can help here.\nIn addition, it makes sense to name the functions provided in the specification in detail. The message structure provides several possible fields: event: determines the type of the message and allows differentiated event handlers in the client; data: contains the actual message body and can be continued across multiple lines, with the client automatically combining the contents into a data block; id: allows each message to be uniquely labeled so that the Last-Event-ID header can be seamlessly continued when the connection is re-established; retry: signals to the client after which time an automatic connection should be established in case the connection is interrupted; finally, comments can be inserted via a leading colon: which are ignored by the client but used for keep-alive mechanisms. These fields together form the normative framework that every compliant SSE server should meet.\nOverall, SSE is suitable for a wide range of real-time scenarios, as long as the inherent limitations are taken into account and compensated for by appropriate architectural decisions. The technology offers an elegant solution for cases where simplicity, standards compliance, and stable, server-side push communication are paramount.\n","date":"5 September 2025","externalUrl":null,"permalink":"/posts/real-time-in-focus-server-sent-events-in-core-java-without-frameworks/","section":"Posts","summary":"Chapter 1 – Introduction # 1.1 Motivation: Real-time communication without polling # In modern applications, it is often necessary to provide new information to the client as quickly as possible. Classic polling , i.e. regularly querying a REST endpoint, is inefficient: it generates unnecessary network traffic and puts a load on both server and client, as requests continue even when there is no new data.\n","title":"Real-Time in Focus: Server-Sent Events in Core Java without Frameworks","type":"posts"},{"content":"","date":"5 September 2025","externalUrl":null,"permalink":"/tags/serverside/","section":"Tags","summary":"","title":"Serverside","type":"tags"},{"content":"","date":"5 September 2025","externalUrl":null,"permalink":"/tags/sse/","section":"Tags","summary":"","title":"Sse","type":"tags"},{"content":"Reactive streams address a fundamental problem of modern systems: Producers (sensors, services, user events) deliver data at an unpredictable rate , while consumers (persistence, UI, analytics) can only process data at a limited speed. Without a flow control model , backlogs, storage pressure, and ultimately outages occur.\nWith Java, java.util.concurrent.Flow provides a minimalist API that standardises this problem: Publisher → Processor → Subscriber, including backpressure. Flow.Processor\u0026lt;I,O\u0026gt; is the hinge between upstream and downstream: process, transform, buffer, throttle , and at the same time correctly respect the demand.\nAbstract: Reactive Streams = asynchronous push model with backpressure. Java Streams = synchronous PullModel without backpressure.\n1.1 Why Reactive Streams? Where does that fit in the JDK? 1.2 Push vs. Pull: Reactive Streams vs. Java Streams 1.3 Typical Use Cases 2. Overview: java.util.concurrent.Flow 2.1 Life cycle of the signals 2.2 Backpressure basic principle 3. Focus on Flow.Processor\u0026lt;I,O\u0026gt; 3.1 Contract \u0026amp; Responsibilities Type Variables \u0026amp; Invariants Chain position: between upstream \u0026amp; downstream 4. Practical introduction with the JDK 4.1 SubmissionPublisher in a nutshell 4.2 Simple pipeline without its own processor 4.3 Limitations of SubmissionPublisher 5. Sample Processor with Code Examples 5.1 MapProcessor\u0026lt;I,O\u0026gt; Transformation 5.2 FilterProcessor – Filtering Data 5.3 BatchingProcessor – Collect and Share 5.4 ThrottleProcessor Throttling 6. Backpressure in practice 6.1 Strategies for dealing with backpressure 6.2 Demand Policy: Batch vs. Single Retrieval 6.3 Monitoring, Tuning and Capacity Planning Concurrency \u0026amp; Execution Context 7.1 Executor Strategies 7.2 Reentrancy traps and serialization of signals 7.3 Virtual Threads (Loom) vs. Reactive Error \u0026amp; Completion Scenarios 8.1 Semantics of onError and onComplete 8.2 Restarts and Retry Mechanisms 8.3 Idem potency and exactly-once processing Interoperability \u0026amp; Integration 9.1 Integration with Reactive Frameworks 9.2 Use in classic applications 9.3 Bridge to Java Streams Result 1.1 Why Reactive Streams? # Controlled load (backpressure): Consumers actively signal how many items they can process (request(s)) by requesting the exact number of items that they can currently process safely via their subscription. The publisher may only deliver as many items as previously requested, maintaining a controlled processing speed, ensuring buffers do not overflow, and keeping memory consumption predictable. The demand can be adjusted dynamically (e.g. batch request(32) or item-wise request(1)). In case of overload, the consumer can pause or cancel cleanly by calling cancel(). Asynchrony \u0026amp; Decoupling: Data flows can cross thread and service boundaries. This means that a publisher is operated in its own thread or via an executor, for example, while the subscriber processes in a different context. In between, there can be queues or network boundaries, for example, when events are sent from one microservice to another. Loose coupling allows systems to scale and distribute without processing being tightly tied to a single execution context. Infinite/long flows: Telemetry, logs, and EventStreams are often continuous, potentially infinite data sources. A single pull, as is standard in classic Java streams, is insufficient here because the data is not available at a fixed time, but rather constantly accumulates. Instead, you need a permanent subscription that continuously informs the consumer about new events. This is the only way to process continuous data streams without the consumer having to poll or repeatedly start new streams actively. Error semantics \u0026amp; completion: Consistent signals such as onError and onComplete ensure that each data pipeline receives a well-defined closure or transparent error handling. Instead of unpredictable exceptions that occur somewhere in the code, there are standardised callback methods that clearly mark the lifetime of a stream. This enables the reliable distinction between a data stream that has been terminated regularly and one that has encountered an error, allowing downstream components to react accordingly, such as through cleanup, retry mechanisms, or logging. This predictability is crucial for robust pipelines that are to run stably over more extended periods of time. Composability: Composability plays a significant role: A processor can be used as a transformation, for example, by modifying incoming data according to specific rules – similar to a map or filter operation in Java streams. In addition, control operators can be implemented, such as a throttle that limits the rate or a debounce, which only passes on the last signal of a short period of time. Finally, a processor also enables the construction of more complex topologies, such as FanIn, where several sources are merged, or FanOut, where a data stream is distributed among multiple subscribers. This allows entire processing pipelines to be assembled in a modular manner without the individual components having to know their internal logic. **Operation \u0026amp; Security: **Backpressure is not only used for load control, but also acts as effective protection against denial‑of service attacks. A system that writes data to infinite buffers without restraint can easily be paralysed by excessive input. With correctly implemented backpressure, on the other hand, every demand is strictly enforced: The publisher only delivers as many elements as were actually requested. This prevents malicious or faulty sources from overloading the system by producing data en masse. Instead, resource consumption remains predictable and controlled, increasing the stability and security of the entire pipeline. Where does that fit in the JDK? # Namespace : java.util.concurrent.Flow – this package bundles the core building blocks of the reactive API. It provides interfaces for publishers, subscribers, subscriptions, and processors, and thus forms the standard framework for reactive streams in the JDK. This ensures that all implementations adhere to the same contracts and can be seamlessly integrated. Roles : The roles can be distinguished as follows: A publisher creates and delivers data to its subscribers, a subscriber consumes this data and processes it further, the subscription forms the contract between the two and regulates the backpressure signals, among other things, and the processor takes on a dual role by receiving data as a subscriber and at the same time passing on transformed or filtered data to the next consumer as a publisher. The types in the Flow API are generic and require careful and thread-safe handling. Generic means that Publisher, Subscriber, and Processor are always instantiated with specific type variables, so that the data type remains consistent throughout the pipeline. At the same time, strict attention must be paid to thread safety during implementation, as signals such as onNext or onError can arrive from different threads and must be synchronised correctly. As an introduction, the JDK provides a reference implementation with the SubmissionPublisher, which is sufficient for simple scenarios and already takes into account essential concepts such as backpressure. In production systems, however, developers often fall back on their own or adapted processor classes to precisely implement specific requirements for buffering, transformation, error handling or performance. 1.2 Push vs. Pull: Reactive Streams vs. Java Streams # Aspect Java Streams (Pull) Reactive Streams / Flow (Push) **Data direction Consumer pulls data Producer pushes data Execution type. synchronously in the calling thread asynchronous, often via executor/threads **Backpressure non-existent central concept (request(s)) Lifetime finite, terminal operation Potentially infinite, subscription-based **Error Exceptions in the caller onError‑signal in the stream Demolition End of Source cancel() of the subscription Minimal examples\nPull (Java Streams) – the consumer sets the pace:\nNote: SubmissionPublisher is a convenient way to get started, but not a panacea (limited tuning options). Custom processor‑implementations give you full control over backpressure, buffer, and concurrency.\n@Test void test001() { var data = List.of(1, 2, 3, 4, 5); int sum = data.stream() .filter(n -\u0026gt; n % 2 == 1) .mapToInt(n -\u0026gt; n * n) .sum(); System.out.println(\u0026#34;sum = \u0026#34; + sum); } //Push (flow) – the producer delivers, the subscriber signals demand: import java.util.concurrent.Flow.*; import java.util.concurrent.SubmissionPublisher; @Test void test002() { class SumSubscriber implements Flow.Subscriber\u0026lt;Integer\u0026gt; { private Flow.Subscriptions; private int sum = 0; @Override public void onSubscribe(Flow.Subscription s) { this.s = s; s.request(1); } @Override public void onNext(Integer item) { sum += item; s.request(1); } @Override public void onError(Throwable t) { t.printStackTrace(); } @Override public void onComplete() { System.out.println(\u0026#34;sum = \u0026#34; + sum); } } try (var pub = new SubmissionPublisher\u0026lt;Integer\u0026gt;()) { pub.subscribe(new SumSubscriber()); for (int i = 1; i \u0026lt;= 5; i++) pub.submit(i); asynchronous push } } 1.3 Typical Use Cases # Telemetry \u0026amp; Logging: In modern systems, events are continuously created, such as metrics or log entries. These streams can be so high frequency that they cannot be permanently stored or transmitted in their raw form. This is where a processor comes into play, which first caches the events and then combines them in batches, for example, 100 events each. In this way, the flood of data can be effectively throttled and persisted in manageable portions, preventing overloading of individual components. Message ingestion: In many architectures, message and event systems, such as Kafka, classic message queues, or HTTP endpoints, form the input layer for data. These can be modelled as publishers who continuously provide new events. A processor then takes over the validation and possible enrichment, such as checking mandatory fields, adding additional metadata or transforming them into a uniform format. Only then does the enriched and checked data pass on to the subscriber, who takes over the actual persistence, for example, by writing it to a database or data warehouse. In this way, a clear separation of responsibility is achieved, allowing errors to be detected and addressed early in the data stream. IoT/streams from sensors: In practice, sensors often deliver data streams at highly fluctuating speeds. Some sensors transmit data in millisecond intervals, while others transmit data only every few seconds or minutes. If these flows are passed on unchecked to a central consumer, there is a risk of buffer overflow and thus instability in the overall system. With backpressure, however, the consumer can actively control how many measured values they take. In scenarios where the rate exceeds the processing capacity, additional strategies can be employed: either older values are discarded (DropStrategy) or only the most recent value is passed on (LatestStrategy). This keeps the system stable, allowing the consumer to always work with up-to-date and relevant data without being overwhelmed by data avalanches. UIEvents \u0026amp; RateControl: In user interfaces, event streams with very high frequencies quickly arise, for example, when a user types a search query and an event is triggered for every keystroke. Without regulation, these events would be passed on to the BackendService unhindered, which could overload both the network connections and the servers. In this case, a special DebounceProcessor ensures that only the last event is forwarded within a short period of time. This avoids unnecessary inquiries, and the user still receives up-to-date suggestions promptly. This technique effectively prevents the UI from being flooded with data and, at the same time improves the perceived performance and responsiveness of the application. Security: For safety-critical applications, it is imperative that each stage of a data pipeline is clearly delineated and only communicates via defined contracts. This prevents the creation of an unbridled shared state that could lead to inconsistencies or security gaps. Another aspect is the possibility of interrupting data flows cleanly at any time. A cancel() method allows a subscriber to terminate the subscription if they determine that the source is faulty, provides too much data, or presents a potential security threat. In this way, resources can be freed up immediately, and the pipeline remains stable and resilient even under adverse conditions. 2. Overview: java.util.concurrent.Flow # After explaining the motivation and the basic differentiation from Java Streams in the first chapter, we will now deal with the actual framework that the JDK provides for reactive streams. With the java.util.concurrent.The Flow package, a small but powerful API, has been provided since Java 9, describing the essential roles and contracts of Reactive Streams. These interfaces are deliberately kept minimal, allowing them to be used directly for simple use cases as well as forming the basis for more complex libraries.\nThe focus is on four roles that define the entire lifecycle of publishers and subscribers: Publisher , Subscriber , Subscription and Processor.\nA publisher is the source of the data. He produces elements and makes them accessible to his subscribers. He may only send data if it has been explicitly requested beforehand. This prevents him from sending events uncontrollably, which in the worst case, fizzle out into nowhere or cause memory problems.\nA subscriber is the counterpart to this: they receive the data that the publisher provides and process it further. To avoid being overloaded, the subscriber informs the publisher of the number of items they can include at a given time as part of the subscription.\nThe subscription is the link between the publisher and the subscriber. It is created at the moment of registration (the so-called subscribe process) and handed over to the subscriber. From then on, it controls the flow of data by providing methods such as request(long n) and cancel(). This controls demand and, simultaneously, enables a clean interruption of data transmission.\nFinally, a processor combines both roles: it is a subscriber itself because it receives data from a publisher, but at the same time, it is also a publisher because it passes on the transformed or filtered data. This dual function is the great strength of the processor: it enables the construction of complex pipelines from simple building blocks, where transformation, filtering, or aggregation are clearly separated and encapsulated.\n2.1 Life cycle of the signals # The flow within the Flow API follows a clear order. As soon as a subscriber logs in to a publisher, the publisher first calls the subscriber\u0026rsquo;s onSubscribe method and passes the subscription. Only when the subscriber requests elements via this subscription – such as request(1) or request(10) – does the publisher begin to deliver data. Each delivered item is delivered by a call to onNext.\nIf all data has been sent or the stream is terminated for other reasons, the publisher signals the end with onComplete. If, on the other hand, an error occurs, it is reported via onError . This precise signal semantics ensures that every data flow is either completed regularly or terminated with an error message. A subscriber, therefore, always knows exactly what state the data source is in and can react accordingly.\n2.2 Backpressure basic principle # A central concept of Flow is the so-called backpressure. It prevents publishers from flooding their subscribers with data. Instead, the subscribers themselves control how many items they want to retrieve. For example, a subscriber can initially request only a single element with request(1) and only ask for the next one after it has been processed. Similarly, it is possible to order larger quantities, such as request(50), to increase throughput. This model gives consumers complete control and ensures a stable balance between production and consumption.\nBackpressure is not only a technical detail, but a decisive criterion for the robustness of reactive systems. Without this control, publishers could deliver events unchecked, overloading resources such as CPU or memory. Backpressure, on the other hand, can also be used to implement complex scenarios such as batching, prioritisation or flow control across thread and system boundaries.\nThe Flow API in the JDK defines the elementary building blocks of a reactive pipeline. Publisher, subscriber, subscription and processor together form a clear model that supports both small experiments and upscaled systems. The signal lifecycle and backpressure mechanisms ensure that the data streams are controlled, traceable and stable. This lays the foundation for delving deeper into the role of the processor in the following chapters and shedding light on its concrete applications.\n3. Focus on Flow.Processor \u0026lt;I,O\u0026gt; # After the four roles were briefly described in the previous chapter, the focus now shifts to the processor. It occupies a key position in the flow API because it fulfils two roles simultaneously: it is a subscriber to its upstream, i.e., it receives data from a publisher, and it is also a publisher to its downstream, i.e., it passes on transformed or filtered data to other subscribers. This makes it act like a hinge in a chain of processing steps.\nA processor is always provided with two type parameters: \u0026lt;I, O\u0026gt;. The first type describes which elements it receives from the upstream, and the second type describes which elements it passes on to the downstream. This allows any transformations to be mapped – for example, from raw sensor data to validated measured values or from text messages to structured objects.\n3.1 Contract \u0026amp; Responsibilities # Implementing a processor means that you must abide by the rules of both subscribers and publishers. This includes, in particular, the following aspects:\nPay attention to signal flow: The processor must handle all signals it receives from the upstream device wholly and correctly. These include the onSubscribe, onNext, onError, and onComplete methods. Each of these signals fulfils a particular function in the lifecycle of a data stream: onSubscribe initiates the subscription and hands over control of the data flow, onNext signals the actual data, onError signals the occurrence of an error, and onComplete marks the regular end of the stream. It is essential that the processor strictly adheres to these semantics: An error may only be reported once and unambiguously, and no further data may follow after the onComplete event occurs. Only through these clear rules does processing remain consistent, predictable and reliable for all components involved. Respect backpressure: As a subscriber, the processor is obligated to adhere strictly to the rules of the backpressure model. This means that it may only receive and process exactly as many elements as it has previously actively requested from the upstream via its subscription – for example, by making a call such as request(1) or request(10). This prevents him from being overrun by a publisher who is too fast. At the same time, as a publisher, he has the responsibility to pass on this logic unchanged to his downstream. He is therefore not allowed to forward more elements to his subscribers than they have explicitly requested. The processor thus acts as a pass-through for the demand signals, ensuring that the entire chain, from publisher to processor to subscriber, remains in balance and that there are no overloads or data losses. Manage resources cleanly: A processor must also ensure that the resources it manages can be released cleanly at all times. In particular, this includes the fact that subscriptions can be actively terminated, for example, by calling cancel() if a subscriber loses interest or processing is to be interrupted for other reasons. It is equally vital that there are no memory or thread leaks: Open queues must be emptied or closed, background threads must be properly terminated, and executor services must be shut down again. Only through this consistent management can the system remain stable, performant and free of creeping resource damage in the long term. Do not swallow errors: If an error occurs, it must be consistently and transparently passed on to the downstream system via the onError method. This ensures that downstream subscribers are also clearly informed about the status of the data stream and can react – for example, through logging, retry mechanisms or the targeted termination of their own processing. If, on the other hand, an error is tacitly ignored, there is a risk of inconsistencies that are difficult to understand. Equally problematic is the uncontrolled throwing of unchecked exceptions, as they are not caught cleanly in the context of reactive pipelines; thus, entire processing chains can enter an undefined state. Clean error propagation is therefore a central quality feature of every processor implementation. Type Variables \u0026amp; Invariants # The type parameters can be used to determine precisely what kind of data a processor can process. For example, a processor\u0026lt;string, integer\u0026gt; could receive lines of text and extract numbers from them, which it then passes on. These types must be strictly adhered to a subscriber who expects integers must not receive strings unexpectedly. The genericity in the Flow API ensures that errors of this kind are already noticeable during compilation.\nAdditionally, the invariant holds that a processor must behave like both a correct subscriber and a correct publisher. He is therefore not only responsible for the transformation, but also for the proper transfer of tax information, such as demand and cancellation.\nChain position: between upstream \u0026amp; downstream # In practice, a processor is rarely used in isolation, but as an intermediate link in a chain. An upstream publisher delivers data that the processor receives, transforms or filters and then forwards to downstream subscribers. This creates flexible pipelines that can be expanded or exchanged depending on requirements.\nFor example, a logging processor can be placed between a data source and the actual consumer, also to record all elements. A processor can also be used to buffer data before it is passed on in larger batches. The position in the middle allows different aspects, such as transformation, control and observation, to be separated from each other and made modular.\nThe Flow.Processor is the link in reactive pipelines. Through his dual function as subscriber and publisher, he takes responsibility for the correct processing of signals, the implementation of transformations and compliance with the backpressure rules. Its type parameters \u0026lt;I,O\u0026gt; also ensure that data flows remain strictly typed and thus reliable. In the following chapters, we will show how to implement your own processor classes and which patterns have proven themselves in practice.\n4. Practical introduction with the JDK # Now that the concepts and contracts around the Flow.Processor have been presented in detail, it is time to start with a practical implementation in the JDK. Fortunately, Java has provided a reference implementation for a publisher since version 9, with the SubmissionPublisher, which is ideal for initial experiments. It relieves you of many details, such as thread management and internal buffering, allowing you to concentrate fully on the data flow.\nThe SubmissionPublisher is designed to distribute data asynchronously to its subscribers. In the background, it works with an executor that packages and distributes the individual signals, such as onNext, onError and onComplete, into separate tasks. By default, it uses the ForkJoinPool.commonPool(), but it can also pass its own executor. For small experiments, the standard configuration is usually sufficient; however, in production scenarios, it is worthwhile to adapt the executor and buffer sizes to your specific requirements.\n4.1 SubmissionPublisher in a nutshell # The SubmissionPublisher class implements the Publisher interface and is therefore an immediately usable source of data. It provides the submit(T item) method, which is used to add new items to the stream. These items are then automatically distributed to all registered subscribers. In this way, the publisher assumes the role of an asynchronous dispatcher, passing incoming data to any number of subscribers.\nAn essential detail is that the submit method does not block, but buffers the elements internally and then distributes them. However, if the buffer limits are reached, submit can block or even throw an IllegalStateException if the system is under too much load. That\u0026rsquo;s why it\u0026rsquo;s crucial to find the right balance between publisher speed, buffer size, and subscriber demand.\nimport java.util.concurrent.Flow.*; import java.util.concurrent.SubmissionPublisher; @Test void test003() { class PrintSubscriber implements Flow.Subscriber\u0026lt;Integer\u0026gt; { private Flow.Subscription subscription; @Override public void onSubscribe(Flow.Subscription subscription) { System.out.println(\u0026#34;Register: 1 element/req\u0026#34; ); this.subscription = subscription; subscription.request(1); Request first element } @Override public void onNext(Integer item) { System.out.println(\u0026#34;Receive: \u0026#34; + item); subscription.request(1); request another after each item } @Override public void onError(Throwable t) { t.printStackTrace(); } @Override public void onComplete() { System.out.println(\u0026#34;Done!\u0026#34;); } } try (SubmissionPublisher\u0026lt;Integer\u0026gt; publisher = new SubmissionPublisher\u0026lt;\u0026gt;()) { publisher.subscribe(new PrintSubscriber()); for (int i = 1; i \u0026lt;= 5; i++) { System.out.println(\u0026#34; send element = \u0026#34; + i ); publisher.submit(i); } } } 4.2 Simple pipeline without its own processor # To understand how SubmissionPublisher works, it\u0026rsquo;s worth taking a simple example. This creates a publisher that sends numbers from 1 to 5 to a subscriber. The subscriber receives the data and outputs it to the console. This example shows the basic flow of onSubscribe, onNext, and onComplete.\nIn this program, the publisher asynchronously generates five values, which are transmitted to the subscriber one after the other. The subscriber explicitly requests an additional element each time, creating a controlled and stable flow of data. Finally, the publisher signals with onComplete that no more data will follow.\n4.3 Limitations of SubmissionPublisher # Even though the SubmissionPublisher is very helpful for getting started, it quickly reaches its limits in more complex scenarios. For example, it offers only limited possibilities for configuring the backpressure behaviour. While the buffer size is customizable, it doesn\u0026rsquo;t directly support more complex strategies, such as dropping or latest-value. Additionally, the standard executor is not suitable for all applications, particularly when high latency requirements or strict thread isolation are necessary.\nAnother point is that SubmissionPublisher is primarily intended for learning purposes and simple applications. In productive systems, people usually rely on their own publisher and processor implementations or on established reactive frameworks such as Project Reactor or RxJava, which are based on the Flow API and provide additional operators.\nThe SubmissionPublisher is a handy place to start to see the concepts of the Flow API in action. It shows how publishers and subscribers interact, how backpressure works and how signals are processed. At the same time, however, it also becomes clear that for more complex or productive scenarios, in-house processor implementations or external libraries are indispensable. In this way, he forms the bridge between the theoretical foundations and the first practical steps in working with reactive pipelines in the JDK.\n5. Sample Processor with Code Examples # Now that the foundations for your own implementation of a processor have been laid, it makes sense to look at typical patterns. They show how concrete use cases can be implemented and form the basis for more complex pipelines. Here are some simple but commonly used processor types, each with sample code.\n5.1 MapProcessor \u0026lt;I,O\u0026gt; Transformation # The MapProcessor is the simplest and at the same time one of the most useful processors. It takes on the role of a transformation: incoming data is changed with a function and then passed on. For example, strings can be converted to their length or numbers can be squared.\n@Test void mapProcessor_usage_minimal() throws Exception { final CountDownLatch done = new CountDownLatch(1); final Function\u0026lt;String, Integer\u0026gt; mapper = s -\u0026gt; Integer.parseInt(s) * 2; //1) Publisher (source) try (SubmissionPublisher\u0026lt;String\u0026gt; publisher = new SubmissionPublisher\u0026lt;\u0026gt;(); //2) MapProcessor (String -\u0026gt; Integer) MapProcessor\u0026lt;String, Integer\u0026gt; map = new MapProcessor\u0026lt;\u0026gt;(mapper)) { //3) Target Subscribers (Spend Only) map.subscribe( new Flow.Subscriber\u0026lt;\u0026gt;() { private Flow.Subscriptions; @Override public void onSubscribe(Flow.Subscription subscription) { this.s = subscription; s.request(1); } @Override public void onNext(Integer item) { out.println(\u0026#34;receive = \u0026#34; + item); s.request(1); } @Override public void onError(Throwable t) { t.printStackTrace(); done.countDown(); } @Override public void onComplete() { done.countDown(); } }); //Concatenation: Publisher -\u0026gt; MapProcessor publisher.subscribe(map); //Send data publisher.submit(\u0026#34;1\u0026#34;); publisher.submit(\u0026#34;2\u0026#34;); publisher.submit(\u0026#34;3\u0026#34;); publisher.close(); //signal end Wait briefly for completion (asynchronous) done.await(); } } /** * Minimal processor that applies a Function\u0026lt;I,O\u0026gt;. * No own threads/executor – uses the defaults of the SubmissionPublisher. */ static class MapProcessor\u0026lt;I, O\u0026gt; extends SubmissionPublisher\u0026lt;O\u0026gt; implements Flow.Processor\u0026lt;I, O\u0026gt; { private final Function\u0026lt;I, O\u0026gt; mapper; private Flow.Subscription subscription; MapProcessor(Function\u0026lt;I, O\u0026gt; mapper) { super(); this.mapper = mapper; } @Override public void onSubscribe(Flow.Subscription subscription) { this.subscription = subscription; subscription.request(1); } @Override public void onNext(I item) { submit(mapper.apply(item)); subscription.request(1); } @Override public void onError(Throwable throwable) { closeExceptionally(throwable); } @Override public void onComplete() { close(); } } This allows data streams to be elegantly transformed without adjusting the rest of the pipeline.\n5.2 FilterProcessor – Filtering Data # Another typical pattern is filtering. Only those elements that meet a certain condition are passed on. This is especially helpful when large amounts of data are generated, of which only a fraction is relevant.\n@Test void filterEvenNumbers() throws Exception { Predicate\u0026lt;Integer\u0026gt; even = n -\u0026gt; { out.println(\u0026#34;Filter: \u0026#34; + n); return n % 2 == 0; }; try (SubmissionPublisher\u0026lt;Integer\u0026gt; publisher = new SubmissionPublisher\u0026lt;\u0026gt;()) { FilterProcessor\u0026lt;Integer\u0026gt; filter = new FilterProcessor\u0026lt;\u0026gt;(even); CountDownLatch done = new CountDownLatch(1); CopyOnWriteArrayList\u0026lt;Integer\u0026gt; received = new CopyOnWriteArrayList\u0026lt;\u0026gt;(); Flow.Subscriber\u0026lt;Integer\u0026gt; subscriber = new Flow.Subscriber\u0026lt;\u0026gt;() { private Flow.Subscription sub; @Override public void onSubscribe(Flow.Subscription subscription) { this.sub = subscription; sub.request(1); } @Override public void onNext(Integer item) { received.add(item); sub.request(1); } @Override public void onError(Throwable throwable) { done.countDown(); } @Override public void onComplete() { done.countDown(); } }; publisher.subscribe(filter); filter.subscribe(subscriber); List.of(1, 2, 3, 4, 5, 6).forEach(publisher::submit); publisher.close(); //Short waiting time for asynchronous processing if (!done.await(1, TimeUnit.SECONDS)) { throw new TimeoutException(\u0026#34;Processing did not finish in time\u0026#34;); } out.println(\u0026#34;received = \u0026#34; + received); assertEquals(List.of(2, 4, 6), received); } } static class FilterProcessor\u0026lt;T\u0026gt; extends SubmissionPublisher\u0026lt;T\u0026gt; implements Flow.Processor\u0026lt;T, T\u0026gt; { private final Predicate\u0026lt;T\u0026gt; predicate; private Flow.Subscription subscription; public FilterProcessor(Predicate\u0026lt;T\u0026gt; predicate) { this.predicate = predicate; } @Override public void onSubscribe(Flow.Subscription subscription) { this.subscription = subscription; subscription.request(1); } @Override public void onNext(T item) { if (predicate.test(item)) { submit(item); } subscription.request(1); } @Override public void onError(Throwable throwable) { closeExceptionally(throwable); } @Override public void onComplete() { close(); } } With this processor, data streams can be reduced in a targeted manner and made more relevant.\n5.3 BatchingProcessor – Collect and Share # Sometimes it doesn\u0026rsquo;t make sense to pass on every single element right away. Instead, they want to collect data and pass it on in blocks. The BatchingProcessor does exactly this job.\nThis processor is useful for processing data efficiently in blocks, for example when writing to a database.\nimport java.util.*; @Test void testBatch() throws InterruptedException { List\u0026lt;List\u0026lt;Integer\u0026gt;\u0026gt; received = new ArrayList\u0026lt;\u0026gt;(); CountDownLatch done = new CountDownLatch(1); try (SubmissionPublisher\u0026lt;Integer\u0026gt; publisher = new SubmissionPublisher\u0026lt;\u0026gt;(); BatchingProcessor\u0026lt;Integer\u0026gt; batching = new BatchingProcessor\u0026lt;\u0026gt;(3)) { publisher.subscribe(batching); batching.subscribe(new Flow.Subscriber\u0026lt;\u0026gt;() { private Flow.Subscription subscription; @Override public void onSubscribe(Flow.Subscription s) { this.subscription = s; s.request(1); } @Override public void onNext(List\u0026lt;Integer\u0026gt; batch) { System.out.println(\u0026#34;onNext - batch = \u0026#34; + batch); received.add(batch); subscription.request(1); } @Override public void onError(Throwable t) { done.countDown(); throw new AssertionError(\u0026#34;Unexpected error\u0026#34;, t); } @Override public void onComplete() { System.out.println(\u0026#34;onComplete...\u0026#34;); done.countDown(); } }); for (int i = 1; i \u0026lt;= 10; i++) { System.out.println(\u0026#34;submitting i = \u0026#34; + i); publisher.submit(i); } publisher.close(); done.await(); } assertThat(received) .containsExactly( List.of(1, 2, 3), List.of(4, 5, 6), List.of(7, 8, 9), List.of(10) ); } class BatchingProcessor\u0026lt;T\u0026gt; extends SubmissionPublisher\u0026lt;List\u0026lt;T\u0026gt;\u0026gt; implements Flow.Processor\u0026lt;T, List\u0026lt;T\u0026gt;\u0026gt; { private final List\u0026lt;T\u0026gt; buffer = new ArrayList\u0026lt;\u0026gt;(); private final int batchSize; private Flow.Subscription subscription; public BatchingProcessor(int batchSize) { this.batchSize = batchSize; } @Override public void onSubscribe(Flow.Subscription subscription) { this.subscription = subscription; subscription.request(1); } @Override public void onNext(T item) { buffer.add(item); if (buffer.size() \u0026gt;= batchSize) { submit(new ArrayList\u0026lt;\u0026gt;(buffer)); buffer.clear(); } subscription.request(1); } @Override public void onError(Throwable throwable) { closeExceptionally(throwable); } @Override public void onComplete() { if (!buffer.isEmpty()) { submit(new ArrayList\u0026lt;\u0026gt;(buffer)); buffer.clear(); } close(); } } 5.4 ThrottleProcessor Throttling # Especially with high-frequency sources, it is necessary to limit the rate of data passed on. A ThrottleProcessor can be implemented in such a way that it only forwards one element at certain time intervals and discards the rest.\n@Test void throttle_allows_items_only_in_interval() throws Exception { try (SubmissionPublisher\u0026lt;Integer\u0026gt; publisher = new SubmissionPublisher\u0026lt;\u0026gt;(); ThrottleProcessor\u0026lt;Integer\u0026gt; throttle = new ThrottleProcessor\u0026lt;\u0026gt;(50)) { // 50 ms interval CollectingSubscriber\u0026lt;Integer\u0026gt; subscriber = new CollectingSubscriber\u0026lt;\u0026gt;(3); we expect 3 elements //Build a pipeline: publisher -\u0026gt; throttle -\u0026gt; subscriber publisher.subscribe(throttle); throttle.subscribe(subscriber); //Send Items: 1 (should through), 2/3 (throttled), 4 (through), 5 (throttled), 6 (through) publisher.submit(1); Thread.sleep(10); publisher.submit(2); Thread.sleep(10); publisher.submit(3); Thread.sleep(60); //\u0026gt; 50ms: Next may go through publisher.submit(4); Thread.sleep(10); publisher.submit(5); Thread.sleep(60); publisher.submit(6); publisher.close(); //Completion of the source assertTrue(subscriber.await(2, TimeUnit.SECONDS), \u0026#34;Timeout while waiting for throttled items\u0026#34;); var received = subscriber.getReceived(); logger().info(\u0026#34;Received {} items\u0026#34;, received); assertEquals(List.of(1, 4, 6), received); } } //---- Processor ---- static class ThrottleProcessor\u0026lt;T\u0026gt; extends SubmissionPublisher\u0026lt;T\u0026gt; implements Flow.Processor\u0026lt;T, T\u0026gt;, HasLogger { private final long intervalMillis; private Flow.Subscription subscription; private long lastEmission = 0; public ThrottleProcessor(long intervalMillis) { this.intervalMillis = intervalMillis; } @Override public void onSubscribe(Flow.Subscription subscription) { this.subscription = subscription; subscription.request(1); } @Override public void onNext(T item) { logger().info(\u0026#34;onNext {}\u0026#34;, item); long now = System.currentTimeMillis(); if (now - lastEmission \u0026gt;= intervalMillis) { logger().info(\u0026#34;submit item \u0026#34; + item); submit(item); lastEmission = now; } else { logger().info(\u0026#34;\u0026lt; intervalMillis - skipping item {}\u0026#34;, item); } subscription.request(1); } @Override public void onError(Throwable throwable) { closeExceptionally(throwable); } @Override public void onComplete() { close(); } } //---- simple collector-subscriber ---- static class CollectingSubscriber\u0026lt;T\u0026gt; implements Flow.Subscriber\u0026lt;T\u0026gt; , HasLogger { private final List\u0026lt;T\u0026gt; received = new ArrayList\u0026lt;\u0026gt;(); private final CountDownLatch latch; private Flow.Subscription subscription; CollectingSubscriber(int expectedCount) { this.latch = new CountDownLatch(expectedCount); } @Override public void onSubscribe(Flow.Subscription subscription) { this.subscription = subscription; subscription.request(Long.MAX_VALUE); unbounded demand } @Override public void onNext(T item) { logger().info(\u0026#34;onNext {}\u0026#34;, item); received.add(item); latch.countDown(); } @Override public void onError(Throwable throwable) { /* no-op for test */ } @Override public void onComplete() { /* no-op for test */ } List\u0026lt;T\u0026gt; getReceived() { return received; } boolean await(long timeout, TimeUnit unit) throws InterruptedException { return latch.await(timeout, unit); } } This pattern can be used, for example, to prevent a UI event stream from generating too many updates uncontrollably.\nThe sample processors presented here are fundamental building blocks for designing reactive pipelines in the JDK. With transformation, filtering, batching and throttling, many typical requirements are covered. The following chapters will focus on implementing backpressure in practice and discuss which strategies have proven successful in operation.\n6. Backpressure in practice # After presenting concrete sample processors in the previous chapter, we will now focus on one of the central topics in the Flow API: Backpressure. While the theory sounds simple – the subscriber determines the number of elements they can process – in practice, it turns out that the right implementation is crucial for stability, efficiency and robustness.\n6.1 Strategies for dealing with backpressure # Backpressure prevents a publisher from flooding its subscribers with data. Nevertheless, there are different strategies for coping with demand:\nBounded buffering: In this strategy, the publisher only keeps a clearly defined, limited number of elements in an internal buffer. Once this buffer is filled, there are several possible reactions: the submit() method can block until there is space again, or it throws an exception to indicate the overload. In this way, an uncontrolled growth of the memory is prevented. This model is particularly suitable in scenarios where a controlled and predictable data rate is required – for example, in systems that are only allowed to process a certain number of requests per second or in hardware interfaces with limited processing capacity. Dropping: When overloaded, new elements are deliberately discarded instead of being cached in an infinite buffer. This strategy makes sense, especially where not every single event is critical and the loss of individual data points remains tolerable. Typical examples are telemetry data, which is generated at a high frequency anyway, or UI events such as mouse movements or keystrokes, where only the current state is relevant for further processing. Dropping keeps the system responsive and resource-efficient even under extreme load, as it doesn\u0026rsquo;t try to funnel every single event through the pipeline at all costs. Latest-Value: In the Latest-Value strategy, old elements are not collected in a buffer, but are consistently discarded as soon as a new value arrives. Only the most recently delivered element is stored in each case, so that the subscriber only receives the latest version the next time it is retrieved or asked. This procedure is beneficial when values change continuously. Still, only the last state is relevant for processing – for example, in cases where sensor data, such as temperature or position coordinates, is involved. This relieves the subscriber because he does not have to work through all the intermediate results, but can continue working directly with the latest information. Blocking: In the blocking strategy, the publisher actively waits for demand to be signalled again by the subscriber before delivering further elements. In practice, this means that the calling thread blocks until a corresponding request is made. This approach is comparatively easy to implement and makes the process calculable at first glance, as it ensures that no more data is produced than was requested. However, this approach has a significant drawback: blocked threads are a scarce resource in highly concurrent systems, and when many publishers block at the same time, entire thread pools can become clogged. Therefore, blocking should only be used in very special scenarios, such as when the data rate is low anyway or when controlling the exact data flow is more important than maximum parallelism. 6.2 Demand Policy: Batch vs. Single Retrieval # Subscribers can determine for themselves how many items they call up at a time. There are two common approaches:\nSingle retrieval ( request(1) ): After each item received, exactly one more is requested. This approach is simple and provides maximum control, but it generates a large number of signals and can become inefficient at very high rates. Batch retrieval ( request(n) ): The subscriber requests several elements at once, such as request(32). This reduces the signal load, allowing the publisher to deliver items in blocks more efficiently. However, the subscriber must then make sure that he can handle these batches. In practice, both approaches are often combined, depending on whether low latency or high throughput is the primary concern.\n6.3 Monitoring, Tuning and Capacity Planning # A decisive success factor in the use of backpressure is monitoring. Without measuring points, it remains unclear whether a system is running stably or is already working at its limits. The following key figures are helpful:\nQueue depth: The queue depth refers to the number of elements currently in the internal buffer that the subscriber has not yet retrieved. It is a direct indicator of whether the publisher is rushing away from the consumer or whether demand is in balance with production. Latency: Latency refers to the time elapsed between the creation of an element and its final processing by the subscriber. It is a key measure of the system\u0026rsquo;s responsiveness: high latency indicates that elements are jamming in buffers or processing steps are taking too long. In contrast, low latency means a smooth data flow. Throughput (items per second): Throughput indicates the number of items that can be processed per unit of time. It\u0026rsquo;s a measure of the entire pipeline\u0026rsquo;s performance: high throughput means that publishers, processors, and subscribers can work together efficiently and meet demand. If the throughput drops below the expected level, this is an indication of a bottleneck – whether due to too small buffers, too slow processing in the subscriber, or insufficient parallelisation. Throughput is therefore an important key figure for realistically estimating the capacity of a system and making targeted optimisations. This data can be used to identify bottlenecks and take appropriate measures, such as adjusting the batch size, increasing the buffers, or introducing additional processor stages.\nConcurrency \u0026amp; Execution Context # A central feature of the Flow API is its close integration with concurrent processing. Unlike classic Java streams, which usually run synchronously in the same thread, reactive streams almost always involve an asynchronous interaction of several threads. Therefore, it is crucial to understand the role of the execution context in detail.\n7.1 Executor Strategies # The Flow API itself does not specify on which threads the signals are processed. Instead, the implementations determine which executor they use. The SubmissionPublisher uses the ForkJoinPool.commonPool() by default, but also allows you to include your own executor. This is extremely important in practice, as the choice of the executor can change the behaviour of the entire system:\nA CachedThreadPool can enable very high parallelism through its dynamic generation and reuse of threads. It grows indefinitely when there are many tasks at the same time, and reduces again when there is less work. This makes it exceptionally flexible and well-suited for unpredictable load peaks. However, this dynamic can also lead to uncontrollable behaviour: If thousands of threads are suddenly created, the context switching load increases significantly, which has an adverse effect on latency. It is therefore less suitable for latency-critical scenarios, as response times can become unpredictable due to the administrative overhead and the potentially high number of threads. A FixedThreadPool works with a fixed number of threads, which are defined when the pool is created. In this way, it creates predictable limits for concurrency and prevents uncontrolled growth of the thread count. This makes it particularly suitable for scenarios in which resources such as CPU cores or memory are to be used in a clearly limited and predictable manner. However, there is a disadvantage when overloaded: If there are more tasks than threads available, queues form. These lead to increasing latencies and can become problematic in critical systems if the queue grows unchecked. It is therefore essential to consciously dimension the pool and possibly use mechanisms such as backpressure or rejection policies to absorb overload situations in a controlled manner. The ForkJoinPool is optimised for fine-grained, recursive, and highly parallelizable tasks. He relies on a work-stealing procedure in which worker threads actively search for functions that are in the queues of other threads when they themselves have no more work. This achieves a high utilisation of the available threads and makes efficient use of computing time. This model excels above all in CPU-intensive calculations, which can be broken down into many small, independent tasks – for example, in divide-and-conquer algorithms or parallelised data analyses. The ForkJoinPool, on the other hand, is less suitable for blocking operations such as network access, file system or database queries. When a worker thread blocks, it becomes unavailable for work stealing, making the pool scarce, and the model’s efficiency suffers. In such cases, it is advisable to outsource blocking work to separate executors with a suitable thread policy or to rely on the virtual threads introduced by Loom, which are more tolerant of blocking operations. Additionally, in specific scenarios, the use of ForkJoinPool is applicable. ManagedBlocker can help to mitigate the adverse effects of blocking operations by allowing the pool size to be temporarily extended. Choosing the right executor is, therefore, a key architectural point that directly impacts throughput, latency, and stability.\n7.2 Reentrancy traps and serialization of signals # Another critical issue is the question of how the onNext, onError and onComplete signals are delivered. The Reactive Streams specification specifies that these signals must be called sequentially and not simultaneously, so they must be serialised. Nevertheless, in practice, parallel calls arise due to your own implementations or clumsy synchronisation. This quickly leads to errors that are difficult to reproduce, the so-called \u0026ldquo;Heisenbugs\u0026rdquo;.\nTherefore, a processor or subscriber must ensure that he is not confronted with onNext calls from several threads at the same time. This can be achieved through synchronisation, using queues or by delegation to an executor, which processes the signals one after the other.\n7.3 Virtual Threads (Loom) vs. Reactive # With Project Loom, Java has introduced a new model: Virtual Threads. These enable the creation of millions of lightweight threads that efficiently support blocking operations. This blurs the classic line between blocking and non-blocking code. The question, therefore, arises: Do we still need reactive streams with complex backpressure at all?\nThe answer is: Yes, but differentiated. While virtual threads make it easier to handle multiple simultaneous operations, they do not automatically resolve the issue of uncontrolled data production. Even with Virtual Threads, a subscriber must be able to specify the number of elements they can process. Backpressure, therefore, remains an important concept. However, virtual threads can help simplify the implementation of subscriber logic, as blocking processing steps now scale much better.\nError \u0026amp; Completion Scenarios # In every reactive data stream, the question arises as to how to deal with errors and the regular closing. The Flow API provides clear semantics for this: Each stream ends either with a successful completion (onComplete) or with an error (onError). This simple but strict model ensures that a subscriber knows exactly what state the data flow is in at all times.\n8.1 Semantics of onError and onComplete # The onError and onComplete methods mark the endpoint of a data stream. Once one of these methods has been called, no further data may be sent through onNext. This contract is crucial because it guarantees consistency: a subscriber can rest assured that they won\u0026rsquo;t have to process any new items at the end.\nonComplete means that all elements have been successfully submitted and the source has been exhausted. Examples include reading a file or playing back a finite list of messages. onError signals that an error occurred during processing. This error is passed on to the subscriber as an exception so that he can react in a targeted manner – for example, through logging, restarts, or emergency measures. It is important to note that onError and onComplete are mutually exclusive. So it must never happen that a stream first breaks off with an error and then signals a regular end.\n8.2 Restarts and Retry Mechanisms # In many use cases, it is not sufficient to simply terminate a stream at the first error. Instead, they want to try to resume processing. Classic examples are network requests or database access, which can fail temporarily.\nThere is no built-in retry functionality in the flow API itself, but it can be implemented at the processor level. For example, after an error, a processor could decide to restart the request or skip the faulty record. It is important to handle the state clearly: A new onSubscribe may only take place when a new data stream is started. For more complex scenarios, patterns such as circuit breaker or retry with exponential backoff are ideal.\nA simple example could be a subscriber who raises a counter on the first error and retries the faulty operation a maximum of three times. Only if an error still occurs after these retries does it finally pass the error on to the downstream:\nclass RetrySubscriber\u0026lt;T\u0026gt; implements Flow.Subscriber\u0026lt;T\u0026gt; { private final Flow.Subscriber\u0026lt;? super T\u0026gt; downstream; private int retries = 0; private static final int MAX_RETRIES = 3; private Flow.Subscription subscription; RetrySubscriber(Flow.Subscriber\u0026lt;? super T\u0026gt; downstream) { this.downstream = downstream; } @Override public void onSubscribe(Flow.Subscription subscription) { this.subscription = subscription; downstream.onSubscribe(subscription); subscription.request(1); } @Override public void onNext(T item) { downstream.onNext(item); subscription.request(1); } @Override public void onError(Throwable t) { if (retries \u0026lt; MAX_RETRIES) { retries++; System.out.println(\u0026#34;Retry \u0026#34; + retries + \u0026#34; after error: \u0026#34; + t.getMessage()); subscription.request(1); Trying to keep going } else { downstream.onError(t); } } @Override public void onComplete() { downstream.onComplete(); } } This simple pattern is of course greatly simplified, but it clarifies the principle of a repeated attempt in the event of temporary errors.\n8.3 Idem potency and exactly-once processing # A central problem with restarts is determining whether operations can be carried out more than once. Many systems are only robust if their operations are idempotent – that is, they produce the same result when executed multiple times. An example is writing a record with a specific key to a database: even if this process is repeated, the final state remains the same.\nIf idempotency is absent, there is a risk of double processing, which can lead to incorrect results or inconsistencies. Therefore, it is good practice to prefer idempotent operations or to ensure that data is not processed multiple times through additional mechanisms, such as transaction IDs, deduplication, or exactly-once processing.\nA simple example: Suppose a processor processes orders and passes each order with a unique ID to an OrderService, which stores it. Even if the same record passes through the processor twice, the ID ensures that it only exists once in the service. This means that the operation remains idempotent.\nA small Java example with a processor and JUnit5Test might look like this:\nimport org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; import java.util.*; import java.util.concurrent.*; import java.util.function.Function; @Test void processor_enforces_idempotence_by_filtering_duplicates() throws Exception { OrderService svc = new OrderService(); try (SubmissionPublisher\u0026lt;Order\u0026gt; pub = new SubmissionPublisher\u0026lt;\u0026gt;()) { DeduplicateProcessor\u0026lt;Order, String\u0026gt; dedup = new DeduplicateProcessor\u0026lt;\u0026gt;(Order::id); pub.subscribe(dedup); dedup.subscribe(new SinkSubscriber(svc)); pub.submit(new Order(\u0026#34;order-1\u0026#34;, \u0026#34;Order A\u0026#34;)); pub.submit(new Order(\u0026#34;order-1\u0026#34;, \u0026#34;Order A (Duplicate)\u0026#34;)); pub.submit(new Order(\u0026#34;order-2\u0026#34;, \u0026#34;Order B\u0026#34;)); pub.close(); Thread.sleep(100); asynchronous drain } assertEquals(2, svc.size()); assertNotNull(svc.get(\u0026#34;order-1\u0026#34;)); assertNotNull(svc.get(\u0026#34;order-2\u0026#34;)); } //Domain record Order(String id, String payload) { } //Processor: filters duplicates based on a key (idempotency in the pipeline) static class DeduplicateProcessor\u0026lt;T, K\u0026gt; extends SubmissionPublisher\u0026lt;T\u0026gt; implements Flow.Processor\u0026lt;T, T\u0026gt; , HasLogger { private final Function\u0026lt;T, K\u0026gt; keyExtractor; private final Set\u0026lt;K\u0026gt; seen = ConcurrentHashMap.newKeySet(); private Flow.Subscription subscription; DeduplicateProcessor(Function\u0026lt;T, K\u0026gt; keyExtractor) { this.keyExtractor = keyExtractor; } @Override public void onSubscribe(Flow.Subscription s) { this.subscription = s; s.request(1); } @Override public void onNext(T item) { logger().info(\u0026#34;Receive {}\u0026#34;, item); if (seen.add(keyExtractor.apply(item))) { logger().info(\u0026#34;no duplicate - submitting\u0026#34;); submit(item); } else { logger().info(\u0026#34;Duplicate - ignore\u0026#34;); } subscription.request(1); } @Override public void onError(Throwable t) { closeExceptionally(t); } @Override public void onComplete() { close(); } } //Subscriber writes to a simple \u0026#34;DB\u0026#34; static class OrderService { private final Map\u0026lt;String, Order\u0026gt; store = new ConcurrentHashMap\u0026lt;\u0026gt;(); void save(Order o) { store.put(o.id(), o); } Order get(String id) { return store.get(id); } int size() { return store.size(); } } static class SinkSubscriber implements Flow.Subscriber\u0026lt;Order\u0026gt;, HasLogger { private final OrderService svc; private Flow.Subscriptions; SinkSubscriber(OrderService svc) { this.svc = svc; } @Override public void onSubscribe(Flow.Subscription s) { this.s = s; s.request(1); } @Override public void onNext(Order item) { logger().info(\u0026#34;Save {}\u0026#34;, item); svc.save(item); s.request(1); } @Override public void onError(Throwable t) { } @Override public void onComplete() { } } The test shows that despite the publication of order-1 being repeated twice, only one instance is sent downstream, thanks to DeduplicateProcessor. The idempotency logic is thus in the processor , not in the subscriber.\nInteroperability \u0026amp; Integration # A key feature of the Flow API is its openness to integration. The API itself is minimal and only defines the basic contracts for publisher, subscriber, subscription and processor. This allows it to be used both standalone and combined with other reactive libraries.\n9.1 Integration with Reactive Frameworks # Many established reactive frameworks, such as Project Reactor , RxJava, or Akka Streams, are based on the same fundamental ideas as the Flow API and are closely tied to the official Reactive Streams specification. They offer either direct adapters or interfaces that can be used to integrate publishers, subscribers and, in particular, your own processor implementations. This means that self-developed building blocks can be easily embedded in complex ecosystems without being tied to a specific framework. A central example is the helper class FlowAdapters in the JDK, which provides conversion methods between java.util.concurrent.Flow and org.reactivestreams.Publisher. This enables seamless integration of your own JDK-based publishers or processors with Reactor or RxJava operators without breaking the contracts. In practice, this means that a processor implemented with Flow can be integrated directly into a Reactor flux or fed from an RxJava observable, allowing existing pipelines to be expanded or migrated step by step.\nA simple example shows the use of FlowAdapters. This example can also be secured with a small JUnit5 test:\nimport org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; import java.util.concurrent.*; import java.util.concurrent.Flow.*; import java.util.concurrent.FlowAdapters; public class FlowAdapterTest { @Test void testFlowAdapterIntegration() throws Exception { try (SubmissionPublisher\u0026lt;String\u0026gt; jdkPublisher = new SubmissionPublisher\u0026lt;\u0026gt;()) { // Flow -\u0026gt; Reactive Streams org.reactivestreams.Publisher\u0026lt;String\u0026gt; rsPublisher = FlowAdapters.toPublisher(jdkPublisher); //back to Flow Publisher\u0026lt;String\u0026gt; flowPublisher = FlowAdapters.toFlowPublisher(rsPublisher); List\u0026lt;String\u0026gt; results = Collections.synchronizedList(new ArrayList\u0026lt;\u0026gt;()); flowPublisher.subscribe(new Subscriber\u0026lt;\u0026gt;() { private subscriptions; @Override public void onSubscribe(Subscription s) { this.s = s; s.request(1); } @Override public void onNext(String item) { results.add(item); s.request(1); } @Override public void onError(Throwable t) { fail(t); } @Override public void onComplete() { } }); jdkPublisher.submit(\u0026#34;Hello World\u0026#34;); jdkPublisher.close(); Thread.sleep(100); Wait asynchronously assertEquals(List.of(\u0026#34;Hello World\u0026#34;), results); } } } This test shows that a SubmissionPublisher can be successfully turned into a Reactive Streams publisher and back again via FlowAdapters. The message \u0026ldquo;Hello world\u0026rdquo; reaches the subscriber correctly at the end.\n9.2 Use in classic applications # The Flow API can also be used sensibly in non-reactive applications. For example, data streams from a message queue or a websocket can be distributed to internal services via a SubmissionPublisher. Legacy systems that have previously relied on polling benefit from distributing and processing events in real-time. This enables modern, reactive architectures to be integrated into existing applications incrementally.\nA simple example illustrates the connection with a legacy service:\nimport java.util.concurrent.SubmissionPublisher; import java.util.concurrent.Flow; //Simulates an existing legacy service that was previously addressed via polling class LegacyService { public void process(String msg) { System.out.println(\u0026#34;LegacyService handles: \u0026#34; + msg); } } //Subscriber who passes incoming events to the legacy service class LegacyServiceSubscriber implements Flow.Subscriber\u0026lt;String\u0026gt; { private Flow.Subscriptions; private final LegacyService service; LegacyServiceSubscriber(LegacyService service) { this.service = service; } @Override public void onSubscribe(Flow.Subscription s) { this.s = s; s.request(1); } @Override public void onNext(String item) { service.process(item); s.request(1); } @Override public void onError(Throwable t) { t.printStackTrace(); } @Override public void onComplete() { System.out.println(\u0026#34;Stream completed\u0026#34;); } } public class LegacyIntegrationDemo { public static void main(String[] args) { LegacyService legacy = new LegacyService(); try (SubmissionPublisher\u0026lt;String\u0026gt; pub = new SubmissionPublisher\u0026lt;\u0026gt;()) { pub.subscribe(new LegacyServiceSubscriber(legacy)); //Example: instead of polling, a MessageQueue delivers new messages pub.submit(\u0026#34;Message 1 from MQ\u0026#34;); pub.submit(\u0026#34;Message 2 from MQ\u0026#34;); } } } This small program simulates use in a classic application: Instead of polling, messages from an external source are immediately forwarded to the LegacyService , which can continue to work unchanged. The Flow API serves as a bridge between modern event processing and existing logic.\n9.3 Bridge to Java Streams # A common question is how the Flow API compares to or even connects to classic Java Streams (the Stream API). While streams typically process finite amounts of data in the pull model, flow works with potentially infinite sequences in the push model. Nevertheless, a combination is possible: A processor could cache incoming data and feed it into a Java stream. Conversely, the results of a stream can be brought back into a reactive context via a publisher. This creates a bridge between batch-oriented and event-driven processing.\nA small JUnit5 example demonstrates this bridge: Here, elements are transferred to a stream via a SubmissionPublisher and then summed.\nimport org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; import java.util.concurrent.*; import java.util.*; public class FlowToStreamTest { @Test void testFlowToStreamIntegration() throws Exception { try (SubmissionPublisher\u0026lt;Integer\u0026gt; publisher = new SubmissionPublisher\u0026lt;\u0026gt;()) { List\u0026lt;Integer\u0026gt; buffer = Collections.synchronizedList(new ArrayList\u0026lt;\u0026gt;()); //Subscriber collects data in a list publisher.subscribe(new Flow.Subscriber\u0026lt;\u0026gt;() { private Flow.Subscriptions; @Override public void onSubscribe(Flow.Subscription s) { this.s = s; s.request(1); } @Override public void onNext(Integer item) { buffer.add(item); s.request(1); } @Override public void onError(Throwable t) { t.printStackTrace(); } @Override public void onComplete() { } }); Publish events publisher.submit(1); publisher.submit(2); publisher.submit(3); publisher.close(); //short pause to wait for asynchronous collection Thread.sleep(100); //Transition to Java Stream: Calculate Sum int sum = buffer.stream().mapToInt(i -\u0026gt; i).sum(); assertEquals(6, sum); } } } In this example, the list acts as a buffer and allows the transition from the asynchronous push model (flow) to the synchronous pull model (stream). In this way, both worlds can be elegantly combined. However, the topic is far more extensive and will be examined in detail in a separate article.\nResult # The Flow API in java.util.Concurrent shows how Java provides a clean, standardised basis for reactive data streams. While Java Streams are intended for finite, synchronous pull scenarios, Flow addresses the challenges of continuous, infinite and asynchronous data sources. Core concepts include backpressure , precise signal semantics (onNext, onError, onComplete), and the dual function of the Flow.Processor create the basis for stable, extensible pipelines.\nThe patterns shown – from map and filter processors to batching and throttling – make it clear that robust processing chains can be built with little code. Supplemented by retry strategies, idempotency and interoperability with established reactive frameworks, a toolbox is created that is suitable for simple learning examples as well as for upscaled systems.\nThis makes it clear: If you work with event streams in Java, you can\u0026rsquo;t get around the Flow API. It bridges the gap between classic streams, modern reactive programming and future developments such as virtual threads. It is crucial to consistently adhere to the principles of backpressure, clean use of resources and clear signal guidance. This results in pipelines that are not only functional, but also durable, high-performance and safe.\n","date":"4 September 2025","externalUrl":null,"permalink":"/posts/core-java-flow-processor/","section":"Posts","summary":"Reactive streams address a fundamental problem of modern systems: Producers (sensors, services, user events) deliver data at an unpredictable rate , while consumers (persistence, UI, analytics) can only process data at a limited speed. Without a flow control model , backlogs, storage pressure, and ultimately outages occur.\n","title":"Core Java - Flow.Processor","type":"posts"},{"content":"","date":"4 September 2025","externalUrl":null,"permalink":"/tags/design-pattern/","section":"Tags","summary":"","title":"Design Pattern","type":"tags"},{"content":"","date":"4 September 2025","externalUrl":null,"permalink":"/tags/flow-api/","section":"Tags","summary":"","title":"Flow API","type":"tags"},{"content":" 1. Introduction # 1.1 Motivation: Event-driven updating without polling # In classic web applications, the pull principle still dominates: Clients repeatedly make requests to the server to detect changes. This polling is simple, but it leads to unnecessary load on the server and network side, especially if the data stock changes only sporadically. Server-Sent Events (SSE) is a standardised procedure that allows the server to signal changes to connected clients actively. This avoids unnecessary requests, while updates reach the interface promptly.\n1. Introduction 1.1 Motivation: Event-driven updating without polling 1.2 Objectives and delimitation 1.3 Overview of the demonstration scenario 2. Conceptual Foundations 2.1 Server-Sent Events (SSE) 2.2 REST as a transport channel for payload data 2.3 Vaadin Flow: Server-Side UI Model and Push 3. Architecture of the demonstrator 3.1 Component Overview 3.2 Communication relationships 3.3 Runtime Environment and Assumptions 4. Data and event model 4.1 Record Structure 4.2 Event Types and Semantics 4.3 Sequence numbering and idempotent reloading 5. REST/SSE Server with In-Process CLI 5.1 Operating concept of the CLI 5.2 Append Log and Consistency Considerations 5.3 SSE Broadcast: Trigger on Successful Input 5.4 Optional optimisations 6. Vaadin Flow Integration (without JavaScript) 6.1 Server-Side SSE Client 6.2 UI Synchronisation and Push 6.3 Interaction patterns 6.4 Delta Retrieval and Full Recall 7. Implementation – REST SSE Server 7.1 REST Server 7.2 Entry 7.3 SseHandler 7.4 DataHandler 8. Implementation – Vaadin Flow UI 8.1 DashboardView 8.2 DataClient 8.3 Entry 8.4 SseClientService 8.5 UiBroadcaster 9. Summary 9.1 Evaluation of the \u0026ldquo;Signal-per-SSE, Data-per-REST\u0026rdquo; pattern 9.2 Didactic benefits and reusability 1.2 Objectives and delimitation # This article aims to demonstrate the basic functionality of SSE in interaction with a Vaadin Flow application. The focus is on the separation of signalling and data retrieval: While SSE is used exclusively for notifications, the actual retrieval of the new data is done via REST endpoints. Security aspects such as authentication or authorisation, as well as persistent data storage, are deliberately excluded to emphasise the core idea clearly.\nThe code is under the following URL on github:\nhttps://github.com/Java-Publications/Blog---Vaadin---How-to-consume-SSE-from-a-REST-Service\n1.3 Overview of the demonstration scenario # The demonstrator consists of three components:\nREST/SSE server with CLI input : New data is entered directly from the console during runtime and stored in simple in-memory storage. Each input immediately triggers an SSE signal. SSE signal : The server sends an event after each input that tells the clients that new data is available. Vaadin Flow application : The UI registers as an SSE client, displays notifications about new data and allows users to reload and display them in a targeted manner via REST retrieval. This constellation provides a clear and comprehensible example of integrating SSE into a server-side Java UI framework. The separation between signal and data allows for a robust and easy-to-understand architecture that is suitable for a wide range of real-time scenarios.\n2. Conceptual Foundations # 2.1 Server-Sent Events (SSE) # Server-sent events (SSE) are a standardised procedure for continuously transmitting messages from the server to the client. Communication is unidirectional: the server sends, the client receives. Technically, SSE is based on a persistent HTTP connection with the MIME type text/event-stream. Messages consist of simple lines of text and are interpreted by the browser or a client framework.\nThe advantages of SSE include the simplicity of implementation, automatic client-side reconnect, and good integration into existing HTTP infrastructures. Limitations result from the restriction to UTF-8 text and the lack of possibility of bidirectional communication. However, for pure notifications and status messages, SSE is usually more efficient and robust than more complex alternatives such as WebSockets or Long Polling.\n2.2 REST as a transport channel for payload data # REST-based endpoints have established themselves as the standard for transmitting structured or binary data. They are stateless, easily scalable and benefit from established infrastructure such as caching and monitoring. In this demonstration scenario, REST is used to provide the actual data that the client explicitly requests after a notification via SSE. This separation ensures clear responsibilities: SSE signals that something has changed; REST offers the content.\n2.3 Vaadin Flow: Server-Side UI Model and Push # Vaadin Flow is a server-side Java framework for web application development. The UI logic runs on the server, while the browser only takes care of the display. Changes in server state are transmitted to the client via a synchronised communication channel. For the integration of SSE, it is particularly relevant that Vaadin works with push : Server-side events can be transferred directly to the interface. This allows the reception of SSE signals to be elegantly combined with UI updates without the need for client-side JavaScript.\n3. Architecture of the demonstrator # 3.1 Component Overview # The demonstrator consists of three main components. The first component is the REST/SSE server. It is responsible for providing an endpoint for entering new data, storing this data in memory and simultaneously sending signals to the connected clients via SSE as soon as the data stock changes. In addition, it provides REST endpoints through which the clients can retrieve the current data.\nThe second component is the in-process CLI , which is embedded directly in the server process. It enables manual entry of new data rows via the console during runtime. Each input is immediately stored as a data record in the in-memory. At the same time, this process triggers an event that is forwarded to the connected clients via SSE.\nThe third component is the Vaadin Flow application. It acts as a client that receives the signals sent via SSE and informs users that new data is available. If desired, the application can load this data specifically via the provided REST endpoints and display it directly in the user interface. This covers the entire process from input to signalling to visibility in the UI.\n3.2 Communication relationships # The architecture follows the pattern signal via SSE, data via REST. As soon as the CLI receives a new input, the server stores the entry in memory and sends a corresponding SSE update to all connected clients. The Vaadin application receives this signal, informs the user of the availability of new data, and provides an option for targeted retrieval. Via a subsequent REST request, the Vaadin application requests the data and updates the user interface.\n3.3 Runtime Environment and Assumptions # Some deliberately simplified framework conditions apply to the demonstration. The server and Vaadin application run locally on the same host, but in separate processes and typically on different ports, such as 8080 for the server and 8081 for the UI. This simulates a realistic separation of the systems without the need for a complex infrastructure.\nIn addition, the CORS policy is set in such a way that access is possible without restriction. This open configuration is only for the sake of simplifying the demonstration, as security considerations are not taken into account in this scenario.\nThe data is stored exclusively in the working memory. Each record entered remains available only during runtime and is lost after the process ends. Persistence mechanisms such as databases or file systems are deliberately not used in this example to focus entirely on the interaction of CLI input, SSE signal and REST retrieval.\nThis architecture is deliberately kept simple to demonstrate the functionality of SSE in combination with REST and Vaadin Flow in a clear and comprehensible way.\n4. Data and event model # 4.1 Record Structure # In the demonstration scenario, new data is generated by entering lines of text in the CLI. Each entry is stored in the server in the form of a simple data record. This consists of a consecutive sequence number that is incremented with each input, a timestamp that documents the time of entry, and the actual content, i.e. the text entered by the operator. It is implicitly assumed that both the REST/SSE server and the Vaadin application are based on the same system time. This is the only way to compare the timestamps directly without the need for additional synchronisation mechanisms. This structure is deliberately kept minimalist to ensure traceability and to focus on the interaction of the components.\n4.2 Event Types and Semantics # Communication between server and client takes place via events that are transmitted in SSE format. The focus is on the update event, which is always sent when a new record is added. At a minimum, the current sequence number is transmitted as the data load of this event, so that the client can detect whether there are any new entries for it. Optionally, an init event could also be used, which signals the start state when a new connection is established. Other event types, such as reset or snapshot, can be provided for advanced scenarios, but are not required for demonstration.\n4.3 Sequence numbering and idempotent reloading # The sequence number plays a central role in the consistency between server and client. It enables the retrieval of only the data records added since the last known number via REST. This allows clients to remain correctly synchronised even if they have missed one or more SSE signals. This makes the reload idempotent: a new call with the same since specification always returns the same result set, regardless of how often it is repeated. This principle facilitates fault tolerance and ensures a stable processing chain.\n5. REST/SSE Server with In-Process CLI # 5.1 Operating concept of the CLI # The server process has an integrated command line interface that can be used to enter new records during runtime. Each input corresponds to a single line of text that is taken directly into the in-memory. This simple operating concept allows the data source to be controlled manually and thus trigger targeted events, which are then signalled to the clients via SSE.\n5.2 Append Log and Consistency Considerations # The entries are stored in the form of an append log: Each new record is added to the end of the existing list. The sequential sequence number ensures that the sequence can be clearly determined. This simple structure ensures that all clients can consistently reload the same state when needed. More complex mechanisms, such as transactions or locks, are not required for demonstration.\n5.3 SSE Broadcast: Trigger on Successful Input # As soon as a new record is added to memory, the server immediately generates an SSE event of type update. This event is sent out to all connected clients and contains at least the current sequence number. This enables all registered recipients to recognise that new data is available. The actual content is not transmitted, but is reserved for later retrieval via REST.\n5.4 Optional optimisations # For a demonstration scenario, it is sufficient to forward each new event immediately. However, in more realistic environments, optimisations can be helpful. This includes, for example, combining several fast inputs into a combined update signal to reduce the number of messages transmitted. Similarly, regular ping comments can be used to stabilise the connection and ensure that inactive clients are detected and removed. These measures increase robustness, but are not necessary for functional demonstration. We will explore these concepts further in another article, specifically when we discuss the open-source URL shortener project.\n6. Vaadin Flow Integration (without JavaScript) # 6.1 Server-Side SSE Client # The Vaadin Flow application operates a server-side SSE client that is permanently connected to the SSE endpoint of the REST server. This allows the application to receive signals independently of client-side JavaScript and process them directly on the server side. If a connection is interrupted, a reconnect attempt is automatically started so that the UI remains continuously informed about the current status.\n6.2 UI Synchronisation and Push # Since Vaadin Flow works on the principle of a server-side UI model, external events must be integrated into the UI thread. This is done via synchronised accesses, which ensure that changes are executed consistently and thread-safe. In combination with Vaadin Push, incoming SSE signals can be converted directly into visible updates. Users notice the update immediately, without the need for a manual refresh. We deliberately disregard the special case of a complete page reload at this point.\n6.3 Interaction patterns # The Vaadin interface is designed to only display a hint at first, rather than automatically loading all new data upon arrival of an incoming SSE signal. This can be done in the form of a notification or by activating a button. Only through a conscious action by the user is the new data retrieved via REST and integrated into the interface. In addition, the user can selectively decide which detailed data is of interest. This interaction pattern illustrates the separation of signalling and data retrieval and makes the functionality particularly clear for demonstration purposes.\n6.4 Delta Retrieval and Full Recall # When reloading the data via REST, a distinction can be made between two modes: A complete retrieval always loads the entire data set. In contrast, a delta retrieval retrieves only those entries that have been added since the last known sequence number. If no sequence number is available, the entire database is not transferred, but only the last n messages are loaded. This procedure is sufficient for the demonstration, as it facilitates traceability. In scenarios closer to production, delta retrieval still offers advantages in terms of efficiency and bandwidth.\n7. Implementation – REST SSE Server # 7.1 REST Server # The RestServer bundles the entire server functionality based on com.sun.net.httpserver.HttpServer. It initialises the three endpoints of the demonstrator (/sse, /data, /health) and manages the server resources required for SSE. These include the number of currently connected output streams (sseClients), an in-memory append log of the data records, and the execution services required at runtime (a Scheduled Executor for Keep pings and a Connection Executor for long-lived SSE connections).\nDuring operation, the server starts a CLIThread that receives rows from stdin and inserts them into the log as new records. Each entry is given a monotonic sequence number and a time stamp; an SSEUpdate is then sent to all connected clients. The RestServer also provides helper functions for CORSHeader, standardised answers, query parsing, and the output of the SSE form. The since(long) and lastN(int) methods map the REST reading paths: They either return all entries after a specific sequence number or the last n entries for initial synchronisation if no sequence is yet known.\npackage com.svenruppert.rest; public class RestServer { public static final int DEFAULT_LAST_N = 20; public static final String PATH_SSE = \u0026#34;/sse\u0026#34;; public static final String PATH_DATA = \u0026#34;/data\u0026#34;; public static final String PATH_HEALTH = \u0026#34;/health\u0026#34;; public static final String CONTENT_TYPE = \u0026#34;text/plain; charset=utf-8\u0026#34;; public static final int PORT = 8080; public static final long PING_INTERVAL_MILLIS = 30_000L; protected static final String ACCESS_CONTROL_ALLOW_ORIGIN = \u0026#34;Access-Control-Allow-Origin\u0026#34;; private final HttpServer http; private final Set\u0026lt;OutputStream\u0026gt; sseClients = ConcurrentHashMap.newKeySet(); private final CopyOnWriteArrayList\u0026lt;Entry\u0026gt; store = new CopyOnWriteArrayList\u0026lt;\u0026gt;(); private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2); private final ExecutorService connectionExecutor = Executors.newCachedThreadPool(); private final AtomicLong seq = new AtomicLong(0); public RestServer(int port) throws IOException { http = HttpServer.create(new InetSocketAddress(port), 0); http.createContext(PATH_SSE, new SseHandler(this)); http.createContext(PATH_DATA, new DataHandler(this)); http.createContext(PATH_HEALTH, ex -\u0026gt; respond(ex, 200, \u0026#34;OK\u0026#34;)); http.setExecutor(connectionExecutor); } ---Main--- public static void main(String[] args) throws Exception { RestServer srv = new RestServer(PORT); Runtime.getRuntime().addShutdownHook(new Thread(srv::stop)); srv.start(); } public AtomicLong getSeq() { return seq; } public Set\u0026lt;OutputStream\u0026gt; getSseClients() { return sseClients; } public ScheduledExecutorService getScheduler() { return scheduler; } public ExecutorService getConnectionExecutor() { return connectionExecutor; } public void writeEvent(OutputStream os, String event, String data) throws IOException { os.write(sseFormat(event, data)); os.flush(); } public void writeComment(OutputStream os, String comment) throws IOException { os.write((\u0026#34;# \u0026#34; + comment + \u0026#34;\\n\\n\u0026#34;).getBytes(StandardCharsets.UTF_8)); os.flush(); } public byte[] sseFormat(String event, String data) { String msg = \u0026#34;event: \u0026#34; + event + \u0026#34;\\n\u0026#34; + \u0026#34;data: \u0026#34; + data + \u0026#34;\\n\\n\u0026#34;; return msg.getBytes(StandardCharsets.UTF_8); } --- Utils --- public void addCors(HttpExchange ex) { ex.getResponseHeaders().add(ACCESS_CONTROL_ALLOW_ORIGIN, \u0026#34;*\u0026#34;); } public void respond(HttpExchange ex, int code, String body) throws IOException { respond(ex, code, body, CONTENT_TYPE); } public void respond(HttpExchange ex, int code, String body, String contentType) throws IOException { Headers h = ex.getResponseHeaders(); h.add(\u0026#34;Content-Type\u0026#34;, contentType); byte[] bytes = body.getBytes(StandardCharsets.UTF_8); ex.sendResponseHeaders(code, bytes.length); try (OutputStream os = ex.getResponseBody()) { os.write(bytes); } } public Map\u0026lt;String, List\u0026lt;String\u0026gt;\u0026gt; parseQuery(URI uri) { Map\u0026lt;String, List\u0026lt;String\u0026gt;\u0026gt; map = new LinkedHashMap\u0026lt;\u0026gt;(); String query = uri.getRawQuery(); if (query == null || query.isEmpty()) return map; for (String pair : query.split(\u0026#34;\u0026amp;\u0026#34;)) { int idx = pair.indexOf(\u0026#39;=\u0026#39;); String key = idx \u0026gt; 0 ? decode(pair.substring(0, idx)) : decode(pair); String val = idx \u0026gt; 0 \u0026amp;\u0026amp; pair.length() \u0026gt; idx + 1 ? decode(pair.substring(idx + 1)) : \u0026#34;\u0026#34;; map.computeIfAbsent(key, k -\u0026gt; new ArrayList\u0026lt;\u0026gt;()).add(val); } return map; } public String decode(Strings) { return URLDecoder.decode(s, StandardCharsets.UTF_8); } public String first(List\u0026lt;String\u0026gt; list) { return (list == null || list.isEmpty()) ? null : list.get(0); } --- Start / Stop --- public void start() { http.start(); System.out.println(\u0026#34;REST/SSE server running on http://localhost:\u0026#34; + PORT); CLI thread for in-process input Thread cli = new Thread(this::cliLoop, \u0026#34;cli-loop\u0026#34;); cli.setDaemon(true); cli.start(); } public void stop() { try { http.stop(0); } catch (Exception ignored) { } try { scheduler.shutdownNow(); } catch (Exception ignored) { } try { connectionExecutor.shutdownNow(); } catch (Exception ignored) { } for (OutputStream os : sseClients) { try { os.close(); } catch (IOException ignored) { } } sseClients.clear(); } ---CLI--- protected void cliLoop() { System.out.println(\u0026#34;CLI ready. Type in lines of text. \u0026#39;exit\u0026#39; terminates the server.\u0026#34;); try (BufferedReader br = new BufferedReader(new InputStreamReader(System.in, StandardCharsets.UTF_8))) { String line; while ((line = br.readLine()) != null) { if (line.equalsIgnoreCase(\u0026#34;exit\u0026#34;)) { System.out.println(\u0026#34;End server...\u0026#34;); stop(); break; } if (line.isBlank()) { System.out.println(\u0026#34;(blank line ignored)\u0026#34;); continue; } appendNew(line); } } catch (IOException e) { System.err.println(\u0026#34;CLI exited: \u0026#34; + e.getMessage()); } } private void appendNew(String text) { long n = seq.incrementAndGet(); Entry e = new Entry(n, Instant.now(), text); store.add(e); System.out.println(\u0026#34;[APPEND] \u0026#34; + e); broadcastUpdate(s); } private void broadcastUpdate(long highestSeq) { String payload = Long.toString(highestSeq); byte[] bytes = sseFormat(\u0026#34;update\u0026#34;, payload); List\u0026lt;OutputStream\u0026gt; dead = new ArrayList\u0026lt;\u0026gt;(); for (OutputStream os : sseClients) { try { os.write(bytes); os.flush(); } catch (IOException e) { dead.add(os); } } if (!dead.isEmpty()) sseClients.removeAll(dead); } public List\u0026lt;Entry\u0026gt; since(long s) { List\u0026lt;Entry\u0026gt; out = new ArrayList\u0026lt;\u0026gt;(); for (Entry e : store) if (e.seq() \u0026gt; s) out.add(e); return out; } public List\u0026lt;Entry\u0026gt; lastN(int n) { int size = store.size(); if (n \u0026lt;= 0) return List.of(); int from = Math.max(0, size - n); return new ArrayList\u0026lt;\u0026gt;(store.subList(from, size)); } } 7.2 Entry # Entry models a single record as a record with the fields seq (sequence number), ts (timestamp) and text (payload from the CLI). The record is immutable and therefore well-suited for concurrent read accesses. The toString() representation is deliberately kept simple and outputs the three fields in one line; it is used for text-based transmission via the /data endpoint and supports simple parsing on the client side.\npackage com.svenruppert.rest; ---Model--- public record Entry(long seq, Instant ts, String text) { @NotNull @Override public String toString() { return seq + \u0026#34;|\u0026#34; + DateTimeFormatter.ISO_INSTANT.format(ts) + \u0026#34;|\u0026#34; + text; } } 7.3 SseHandler # The SseHandler implements the SSE endpoint/sse. When called, it sets up the HTTP response for a text/event stream , inserts the client\u0026rsquo;s output stream into the managed set of SSE connections, and sends an initial event to confirm the successful connection. A periodic Keep‑Alive mechanism (Ping Comments) keeps the connection open and supports the detection of aborted clients. The handler works closely with the helper functions provided by the RestServer (writing individual events or comments, accessing schedulers, and connection management).\nIf there is a write error or the remote station is closed, the handler removes the connection from the set of active streams and cleans up associated resources. In combination with the server\u0026rsquo;s broadcasting, this creates a resilient but straightforward push model for signals via newly available data.\npackage com.svenruppert.rest.handler; ---SIZE--- public record SseHandler(RestServer restServer) implements HttpHandler { @Override public void handle(HttpExchange ex) throws IOException { if (!\u0026#34; GET\u0026#34;.equals(ex.getRequestMethod())) { restServer.respond(ex, 405, \u0026#34;Method Not Allowed\u0026#34;); return; } restServer.addCors(ex); Headers h = ex.getResponseHeaders(); h.add(\u0026#34;Content-Type\u0026#34;, \u0026#34;text/event-stream; charset=utf-8\u0026#34;); h.add(\u0026#34;Cache-Control\u0026#34;, \u0026#34;no-cache\u0026#34;); h.add(\u0026#34;Connection\u0026#34;, \u0026#34;keep-alive\u0026#34;); ex.sendResponseHeaders(200, 0); final OutputStream os = ex.getResponseBody(); restServer.getSseClients().add(os); Initial Event restServer.writeEvent(os, \u0026#34;init\u0026#34;, \u0026#34;ready\u0026#34;); // Keep-Alive Pings ScheduledFuture\u0026lt;?\u0026gt; pinger = restServer.getScheduler().scheduleAtFixedRate(() -\u0026gt; { try { restServer.writeComment(os, \u0026#34;ping\u0026#34;); } catch (IOException e) { /* will be closed below */ } }, RestServer.PING_INTERVAL_MILLIS, RestServer.PING_INTERVAL_MILLIS, TimeUnit.MILLISECONDS); //Keep blocking open until client/OS closes restServer.getConnectionExecutor().execute(() -\u0026gt; { try (os) { //NOP: we only write at events, otherwise Ping will keep the line open //Wait for an exception to occur/OS to close //(an explicit read/write loop is not necessary here) Thread.currentThread().join(); } catch (InterruptedException ignored) { } catch (Exception ignored) { } finally { pinger.cancel(true); restServer.getSseClients().remove(os); try { os.close(); } catch (IOException ignored) { } } }); } } 7.4 DataHandler # The DataHandler provides the REST endpoint/data and implements the two intended read variants. Depending o sequence number is known,on the Query parameters, it returns all entries with a sequence number greater than a given the last n entries by default (configurable via DEFAULT_LAST_N or the lastN parameter). The response is generated as text/plain, with each record output on its own line.\nThe handler also validates the QueryParameters and produces consistent error messages for invalid inputs. Together with the SseHandler, it maps the separation of signaling (SSE) and payload retrieval (REST) and allows a deterministic, idempotent reload process on the client side.\npackage com.svenruppert.rest.handler; ---DATA--- public record DataHandler(RestServer restServer) implements HttpHandler { @Override public void handle(HttpExchange ex) throws IOException { if (!\u0026#34; GET\u0026#34;.equals(ex.getRequestMethod())) { restServer.respond(ex, 405, \u0026#34;Method Not Allowed\u0026#34;); return; } restServer.addCors(ex); Map\u0026lt;String, List\u0026lt;String\u0026gt;\u0026gt; q = restServer.parseQuery(ex.getRequestURI()); String sinceStr = restServer.first(q.get(\u0026#34;since\u0026#34;)); String lastNStr = restServer.first(q.get(\u0026#34;lastN\u0026#34;)); List\u0026lt;Entry\u0026gt; result; if (sinceStr != null \u0026amp;\u0026amp; !sinceStr.isBlank()) { long s; try { s = Long.parseLong(sinceStr); } catch(NumberFormatException nfe) { restServer.respond(ex, 400, \u0026#34;since must be a number\u0026#34;); return; } result = restServer.since(s); } else { int n = RestServer.DEFAULT_LAST_N; if (lastNStr != null \u0026amp;\u0026amp; !lastNStr.isBlank()) { try { n = Integer.parseInt(lastNStr); } catch(NumberFormatException nfe) { restServer.respond(ex, 400, \u0026#34;lastN must be a number\u0026#34;); return; } } result = restServer.lastN(n); } Output as text/plain, one line per entry: seq|ISO-TS|text StringBuilder sb = new StringBuilder(); for (Entry e : result) { sb.append(e.toString()).append(\u0026#39;\\n\u0026#39;); } restServer.respond(ex, 200, sb.toString(), RestServer.CONTENT_TYPE); } } 8. Implementation – Vaadin Flow UI # 8.1 DashboardView # The DashboardView is the central view of the demonstration and is integrated via the route dashboard. It presents the current data in a grid and provides the option to reload with a button. When attached, the view registers with the UiBroadcaster and binds a listener to the SseClientService. If an \u0026ldquo;update\u0026rdquo; signal arrives, the view informs the users (status bar, notification) and activates the reload button. The actual retrieval only takes place after user action: Either new entries are loaded since the last known sequence (lastSeq) or, if no sequence is yet available, the last n messages. After successful retrieval, lastSeq is updated, and the grid is updated consistently. With the detach, the view moves away from the broadcaster and deregisters the listener.\npackage com.svenruppert.flow.views.dashboard; @Route(value = DashboardView.ROUTE, layout = MainLayout.class) public class DashboardView extends Composite\u0026lt;VerticalLayout\u0026gt; implements HasLogger { public static final String ROUTE = \u0026#34;dashboard\u0026#34;; ---Configuration--- private static final String BASE = \u0026#34;http://localhost:8090\u0026#34;; Server Base private static final String SSE_URL = BASE + \u0026#34;/sse\u0026#34;; private static final String DATA_URL = BASE + \u0026#34;/data\u0026#34;; private static final int DEFAULT_LAST_N = 20; ---UI--- private final Grid\u0026lt;Entry\u0026gt; grid = new Grid\u0026lt;\u0026gt;(Entry.class, false); private final Button fetchBtn = new Button(\u0026#34;fetch data\u0026#34;); private final Span status = new Span(\u0026#34;Waiting for events ...\u0026#34;); --- client services --- private final DataClient dataClient = new DataClient(DATA_URL); private final SseClientService sseClient = new SseClientService(SSE_URL); private final AtomicBoolean hasNew = new AtomicBoolean(false); ---Condition--- private volatile Long lastSeq = zero; last confirmed sequence public DashboardView() { getContent().setSizeFull(); grid.addColumn(Entry::seq).setHeader(\u0026#34;Seq\u0026#34;).setAutoWidth(true).setFlexGrow(0); grid.addColumn(e -\u0026gt; e.ts().toString()).setHeader(\u0026#34;Timestamp\u0026#34;).setAutoWidth(true).setFlexGrow(0); grid.addColumn(Entry::text).setHeader(\u0026#34;Text\u0026#34;).setFlexGrow(1); fetchBtn.setEnabled(false); fetchBtn.addClickListener(e -\u0026gt; fetch()); getContent().add(status, fetchBtn, grid); addAttachListener(ev -\u0026gt; { UI ui = ev.getUI(); UiBroadcaster.register(ui); SseClientService.Listener l = (type, data) -\u0026gt; { if (\u0026#34;update\u0026#34;.equals(type)) { try { long seq = Long.parseLong(data.trim()); mark only; Fetch decides on Delta/lastN hasNew.set(true); UiBroadcaster.broadcast(() -\u0026gt; { fetchBtn.setEnabled(true); status.setText(\u0026#34;New data available (seq=\u0026#34; + seq + \u0026#34;) – please retrieve.\u0026#34;); Notification.show(\u0026#34;New data available\u0026#34;, 1200, Notification.Position.TOP_CENTER); }); } catch (NumberFormatException ignore) { Fallback: Hint without seq hasNew.set(true); UiBroadcaster.broadcast(() -\u0026gt; { fetchBtn.setEnabled(true); status.setText(\u0026#34;New data available – please retrieve.\u0026#34;); }); } } }; sseClient.addListener(l); addDetachListener(ev2 -\u0026gt; { sseClient.removeListener(l); UiBroadcaster.unregister(ui); }); }); } private void fetch() { fetchBtn.setEnabled(false); List\u0026lt;Entry\u0026gt; toShow; if (lastSeq != null) { toShow = dataClient.fetchSince(lastSeq); } else { toShow = dataClient.fetchLastN(DEFAULT_LAST_N); } if (!toShow.isEmpty()) { lastSeq = toShow.getLast().seq(); Latest Bookmark } UI ui = UI.getCurrent(); if (ui != null) { effectively final ui.access(() -\u0026gt; { if (!toShow.isEmpty()) { List\u0026lt;Entry\u0026gt; current = new ArrayList\u0026lt;\u0026gt;(grid.getListDataView().getItems().toList()); current.addAll(toShow); grid.setItems(current); status.setText(\u0026#34;Data loaded(\u0026#34; + current.size() + \u0026#34; entries).\u0026#34;); } else { status.setText(\u0026#34;No new entries.\u0026#34;); } hasNew.set(false); }); } } } 8.2 DataClient # The DataClient encapsulates the REST‑communication with the /data endpoint. It offers two read paths: fetchSince(long since) for delta fetches and fetchLastN(int n) for initial synchronisation. The answers are expected as text/plain and converted to a list of Entry (format: seq|ISOTS|text, one line per record). By encapsulating the HTTPDetails, the view remains slim; Changes to the transport format or timeouts can be adjusted centrally without touching the UICode.\npackage com.svenruppert.flow.views.dashboard; --- REST client for /data --- public final class DataClient { private final HttpClient http = HttpClient.newBuilder() .connectTimeout(Duration.ofSeconds(5)).build(); private final String baseUrl; public DataClient(String baseUrl) { this.baseUrl = baseUrl; } //Server format: seq|ISO-TS|text per line private static List\u0026lt;Entry\u0026gt; parse(String body) throws IOException { List\u0026lt;Entry\u0026gt; out = new ArrayList\u0026lt;\u0026gt;(); if (body == null || body.isBlank()) return out; String[] lines = body.split(\u0026#34;\\\\R\u0026#34;); for (String line : lines) { if (line.isBlank()) continue; String[] parts = line.split(\u0026#34;\\\\|\u0026#34;, 3); if (parts.length \u0026lt; 3) continue; long seq = Long.parseLong(parts[0]); Instant ts = Instant.parse(parts[1]); String text = parts[2]; out.add(new Entry(seq, ts, text)); } return out; } public List\u0026lt;Entry\u0026gt; fetchSince(long since) { String url = baseUrl + \u0026#34;?since=\u0026#34; + since; return fetch(url); } public List\u0026lt;Entry\u0026gt; fetchLastN(int n) { String url = baseUrl + \u0026#34;?lastN=\u0026#34; + n; return fetch(url); } private List\u0026lt;Entry\u0026gt; fetch(String url) { try { HttpRequest req = HttpRequest.newBuilder(URI.create(url)) .timeout(Duration.ofSeconds(5)). GET().build(); HttpResponse\u0026lt;String\u0026gt; resp = http.send(req, HttpResponse.BodyHandlers.ofString()); if (resp.statusCode() != 200) return List.of(); return parse(resp.body()); } catch (Exception e) { return List.of(); } } } 8.3 Entry # Entry represents a single record with a monotonic sequence number, timestamp, and message content. The immutable record is well suited for display in the grid and for concurrency in the view. In the Vaadin demonstration, Entry serves as a lightweight transport and display object that is generated directly from the REST response and passed on unchanged.\npackage com.svenruppert.flow.views.dashboard; public record Entry(long seq, Instant ts, String text) { } 8.4 SseClientService # The SseClientService represents the server-side SSEClient of the Vaadin‑application. It maintains a long-lived HTTP‑connection to /sse, parses incoming events in the format text/event-stream and distributes them to registered listeners in the application. After disconnections, a reconnect is automatically attempted. The service‑boundary is deliberately simple: Events are passed on as type/data (especially event: update, data: \u0026lt;seq\u0026gt;). The View then decides whether and how to reload. This preserves the separation between signaling (SSE) and data transmission (REST).\npackage com.svenruppert.flow.views.dashboard; //--- SSE client (server-side) --- public final class SseClientService { private final HttpClient http = HttpClient.newBuilder() .connectTimeout(Duration.ofSeconds(5)).build(); private final String url; private final List\u0026lt;Listener\u0026gt; listeners = new CopyOnWriteArrayList\u0026lt;\u0026gt;(); private volatile boolean running = false; public SseClientService(String url) { this.url = Objects.requireNonNull(url); } private static void sleep(long ms) { try { Thread.sleep(ms); } catch (InterruptedException ignored) { } } public void addListener(Listener l) { listeners.add(l); ensureLoop(); } public void removeListener(Listener l) { listeners.remove(l); } private synchronized void ensureLoop() { if (running) return; running = true; Thread t = new Thread(this::loop, \u0026#34;sse-client-loop\u0026#34;); t.setDaemon(true); t.start(); } private void loop() { while (running) { try { HttpRequest req = HttpRequest.newBuilder(URI.create(url)) .header(\u0026#34;Accept\u0026#34;, \u0026#34;text/event-stream\u0026#34;) .timeout(Duration.ofSeconds(30)). GET().build(); HttpResponse\u0026lt;InputStream\u0026gt; resp = http.send(req, HttpResponse.BodyHandlers.ofInputStream()); if (resp.statusCode() != 200) { sleep(1500); continue; } try (var is = resp.body(); var br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8))) { String line; String event = \u0026#34;message\u0026#34;; StringBuilder data = new StringBuilder(); while ((line = br.readLine()) != null) { if (line.isEmpty()) { // Complete event if (!data.isEmpty()) { fire(event, data.toString()); data.setLength(0); event = \u0026#34;message\u0026#34;; } continue; } if (line.startsWith(\u0026#34;:\u0026#34;)) continue; Comment if (line.startsWith(\u0026#34;event:\u0026#34;)) { event = line.substring(\u0026#34;event:\u0026#34;.length()).trim(); } else if (line.startsWith(\u0026#34;data:\u0026#34;)) { if (!data.isEmpty()) data.append(\u0026#39;\\n\u0026#39;); data.append(line.substring(\u0026#34;data:\u0026#34;.length()).trim()); } } } } catch (Exception e) { sleep(1500); } } } private void fire(String type, String data) { for (listener l : listeners) { try { l.onEvent(type, data); } catch (Exception ignored) { } } } public interface Listener { void onEvent(String type, String data); } } 8.5 UiBroadcaster # The UiBroadcaster is a small helper class that distributes UI updates thread-safely to several active UIs. It maintains a list of currently connected UI instances and executes provided commands within the respective UI context (ui.access(...)). This allows reactions to external signals (SSE) to be safely integrated into the VaadinUI without risking UIThread‑violations. The view registers on the attach and deregisters on the detach, thus avoiding zombie‑references.\npackage com.svenruppert.flow.views.dashboard; //--- Simple broadcaster to distribute UI updates thread-safe --- public final class UiBroadcaster { private UiBroadcaster() { } private static final List\u0026lt;UI\u0026gt; UI_LIST = new CopyOnWriteArrayList\u0026lt;\u0026gt;(); public static void register(UI ui) { UI_LIST.add(ui); } public static void unregister(UI ui) { UI_LIST.remove(ui); } public static void broadcast(Command task) { for (UI ui : List.copyOf(UI_LIST)) { try { ui.access(task); } catch (Exception ignored) { } } } } 9. Summary # 9.1 Evaluation of the \u0026ldquo;Signal-per-SSE, Data-per-REST\u0026rdquo; pattern # The demonstration showed that the separation of signalling and data transmission enables a clear and comprehensible architecture. SSE is ideal for informing clients of changes in real time, while REST takes care of the reliable and flexible delivery of the actual data. This division of labour results in low communication costs and, at the same time, a high level of transparency for users.\n9.2 Didactic benefits and reusability # The scenario presented is deliberately kept simple to make the functionality of SSE in interaction with a Vaadin Flow application understandable. The integration of a CLI for data entry makes it easier to understand the chain of events, signals and retrieval clearly. This makes the example suitable not only for technical experiments, but also for training, workshops and teaching materials. Due to its modular structure, it can be easily extended or integrated into more complex projects, for example, as a basis for further experiments in the context of the URL shortener open source project.\n","date":"3 September 2025","externalUrl":null,"permalink":"/posts/signal-via-sse-data-via-rest-a-vaadin-demonstration-in-core-java/","section":"Posts","summary":"1. Introduction # 1.1 Motivation: Event-driven updating without polling # In classic web applications, the pull principle still dominates: Clients repeatedly make requests to the server to detect changes. This polling is simple, but it leads to unnecessary load on the server and network side, especially if the data stock changes only sporadically. Server-Sent Events (SSE) is a standardised procedure that allows the server to signal changes to connected clients actively. This avoids unnecessary requests, while updates reach the interface promptly.\n","title":"Signal via SSE, data via REST – a Vaadin demonstration in Core Java","type":"posts"},{"content":"","date":"3 September 2025","externalUrl":null,"permalink":"/categories/uncategorized/","section":"Categories","summary":"","title":"Uncategorized","type":"categories"},{"content":" 1. Introduction and motivation # The observer pattern is one of the basic design patterns of software development and is traditionally used to decouple state changes and process them. Its origins lie in the development of graphical user interfaces, where a shift in the data model required synchronising several views immediately, without a direct link between these views. This pattern quickly established itself as the standard solution to promote loosely coupled architectures.\n1. Introduction and motivation 2. The classic observer pattern 3. Observer Pattern in Vaadin Flow 4. Differences between classic Observer and Vaadin Flow 5. When does the classic observer pattern still make sense? 6. Example: Combination of Observer Pattern and Vaadin Flow (with external producer) 7. Best Practices and Pitfalls 8. Conclusion At its core, the Observer Pattern addresses the challenge that events or state changes do not remain isolated; instead, they must be processed by different components. This creates an asynchronous notification model: The subject informs all registered observers as soon as its state changes. The observers react to this individually, without the subject having to know their concrete implementations.\nThe source code with the examples can be found in the following git-repo:https://github.com/Java-Publications/Blog\u0026mdash;Vaadin\u0026mdash;How-to-use-the-Observer-Pattern-and-why** __**\nIn modern software architectures, characterised by interactivity, parallelism, and distributed systems, this basic principle remains highly relevant. It continues to be used in the field of UI development, as user interactions and data flows must be handled dynamically and in a reactive manner. Frameworks such as Vaadin Flow take up this idea, but go beyond the classic implementation by providing mechanisms for state matching, server-side synchronisation and client-side updates.\nThe motivation to apply the Observer Pattern in the context of Vaadin Flow lies in comparing proven yet simple patterns and their further development within a modern UI framework. This not only sharpens the understanding of loose coupling and event handling, but also creates a practical reference to today\u0026rsquo;s web applications.\n2. The classic observer pattern # The classical observer pattern describes the relationship between a subject and a set of observers. The subject maintains the current state and automatically informs all registered observers of any relevant change. This achieves a loose coupling: the subject only knows the interface of its observers, but not their concrete implementation or functionality.\nIn its original form, according to the \u0026ldquo;Gang of Four\u0026rdquo; definition, the pattern consists of two central roles. The subject manages the list of its observers and provides methods of logging in and out. As soon as the internal state changes, it calls a notification method for all observers. The observers implement an interface through which they are informed and can define their own reaction. This allows state changes to be propagated without creating complex coupling or cyclic dependencies.\nTo avoid outdated dependencies and to maintain control over the API, an independent, generic variant with its own interfaces is shown below. The aim is to demonstrate how it works in a JUnit 5 test without the main method. The output is a logger via HasLogger from the package com.svenruppert.dependencies.core.logger.\npackage com.svenruppert.demo; import com.svenruppert.dependencies.core.logger.HasLogger; import org.junit.jupiter.api.Test; import java.util.List; import java.util.concurrent.CopyOnWriteArrayList; import static org.junit.jupiter.api.Assertions.assertEquals; public class ObserverDemoTest { @Test void beobachter_erhaelt_updates_und_letzten_wert() { var sensor = new TemperatureSensor(); var anzeigeA = new TemperaturAnzeige(); var displayB = new Temperature display(); sensor.addObserver(displayA); sensor.addObserver(displayB); sensor.setTemperature(22); assertEquals(22, displayA.lastObservedTemp); assertEquals(22, sensor.getTemperature()); sensor.setTemperature(25); assertEquals(25, sensor.getTemperature()); assertEquals(25, displayA.lastObservedTemp); assertEquals(25, displayB.lastObservedTemp); sensor.removeObserver(displayA); sensor.setTemperature(21); assertEquals(21, sensor.getTemperature()); assertEquals(25, displayA.lastObservedTemp); assertEquals(21, displayB.lastObservedTemp); } /** * Minimalist, own Observer API (generic, thread-safe implemented in the subject). */ interface Observer\u0026lt;T\u0026gt; { void onUpdate(T value); } interface Subject\u0026lt;T\u0026gt; { void addObserver(Observer\u0026lt;T\u0026gt; o); void removeObserver(Observer\u0026lt;T\u0026gt; o); void notifyObservers(T value); } /** * Concrete subject: maintains a temperature and informs observers of changes. */ static class Temperature Sensor implements Subject\u0026lt;Integer\u0026gt; { private final List\u0026lt;Observer\u0026lt;Integer\u0026gt;\u0026gt; observers = new CopyOnWriteArrayList\u0026lt;\u0026gt;(); Private int temperature; @Override public void addObserver(Observer\u0026lt;Integer\u0026gt; o) { observers.add(o); } @Override public void removeObserver(Observer\u0026lt;Integer\u0026gt; o) { observers.remove(o); } @Override public void notifyObservers(Integer value) { observers.forEach(o -\u0026gt; o.onUpdate(value)); } public int getTemperatur() { return temperatur; } public void setTemperature(int newtemperature) { this.temperature = newtemperature; notifyObservers(temperature); } } /** * Concrete Observer: remembers the last value and logs it. */ static class temperature display implements Observer\u0026lt;Integer\u0026gt;, HasLogger { Integer lastObservedTemp; @Override public void onUpdate(Integer value) { lastObservedTemp = value; logger().info(\u0026#34;New temperature: {}°C\u0026#34;, value); } } } In the example, the company\u0026rsquo;s own API defines two lean interfaces: Subject manages observers and propagates changes, while Observer reacts to updates. The implementation of the Subject uses a CopyOnWriteArrayList to handle concurrent logins and logoffs during notification robustly. The JUnit5Test demonstrates the expected behaviour: By setting the temperature twice, notifications are triggered twice. The temperature display logs the values via HasLogger and retains the last observed value for assertion. This demonstrates the basic semantics of the Observer Pattern – loose coupling, clear responsibilities and event-driven updating – in a modern, framework-free Java variant.\n3. Observer Pattern in Vaadin Flow # While the classic Observer Pattern has its origins in desktop programming, Vaadin Flow elevates the concept to a higher level of abstraction. Since Vaadin operates on the server side and manages the UI state centrally, synchronisation between the data model, user interface, and user interactions is an integral part of the framework. Events and listeners take on the role of the observer mechanism.\nA central element is the ValueChangeListener , which is automatically triggered when changes are made to input components such as text fields or checkboxes. Here, the UI component acts as a subject, while listeners represent the observer. As soon as the user makes an input, an event is generated, and all registered listeners are informed. All the developer has to do is implement the desired response without worrying about chaining event flows.\nAdditionally, Vaadin provides a powerful tool with the Binder , which enables bidirectional data binding between the data model and UI components. Changes to a field in the form are automatically propagated to the bound object, while changes in the object are immediately visible in the UI elements. The binder thus adopts a bidirectional observation that goes beyond the possibilities of the classic observer pattern.\nAnother example of the observer pattern\u0026rsquo;s implementation in Vaadin Flow is the EventBus mechanism , which enables components or services to exchange information about events in a loosely coupled manner. Here, it becomes clear that Vaadin adopts the basic idea of the pattern, but expands it in the context of a web application to include aspects such as thread security, lifecycle management, and client-server synchronisation.\nThe strength of Vaadin Flow lies in the fact that developers no longer have to implement observer interfaces explicitly; instead, they can establish observation relationships via declarative APIs, such as addValueChangeListener, addClickListener, or via binder bindings. This does not abolish the pattern, but integrates it deeply into the architecture and adapts it to the special requirements of a server-side web UI.\nVaadin Flow implements the basic idea of the Observer Pattern along two levels. At the UI level, component-specific events (e.g., ValueChangeEvent, ClickEvent) exist that are registered via listeners and trigger changes to the representation or state model. This mechanism corresponds to a finely granulated, component-internal observer and is conveyed via an internal EventBus for each component. At the application level, a domain-level observation structure is also recommended to model state changes outside the UI and to mirror them in the UI in a decoupled manner. This allows long-lived domain objects and short-lived UII punches to be cleanly separated.\nA minimal example exclusively with Vaadin on-board tools shows the observation of UIC conditions as well as the binding to a model via binder:\nimport com.svenruppert.flow.MainLayout; import com.vaadin.flow.component.button.Button; import com.vaadin.flow.component.html.Div; import com.vaadin.flow.component.orderedlayout.VerticalLayout; import com.vaadin.flow.component.textfield.TextField; import com.vaadin.flow.data.binder.binder; import com.vaadin.flow.router.Route; @Route(value = \u0026#34;observer-bordmittel\u0026#34;, layout = MainLayout.class) public class On-board ResourcesView extends VerticalLayout { public BordmittelView() { var name = new TextField(\u0026#34;Name\u0026#34;); var log = new Div(); var save = new Button(\u0026#34;Save\u0026#34;); //(1) Component-internal observer: reacts to ValueChangeEvent name.addValueChangeListener( e -\u0026gt; log.setText(\u0026#34;UI → UI: Name changed to \u0026#39;\u0026#34; + e.getValue() + \u0026#34;\u0026#39;\u0026#34;)); //(2) Binder as a bidirectional observation between UI and model var binder = new Binder\u0026lt;\u0026gt;(Person.class); var model = new Person(); binder.forField(name).bind(Person::getName, Person::setName); binder.setBean(model); // (3) Action: reads current model state (powered by the binder) store.addClickListener( e -\u0026gt; log.setText(\u0026#34;Model → Action: saved name = \u0026#34; + model.getName())); add(name, store, log); } static class Person { private String name; public String getName() { return name; } public void setName(String name) { this.name = name; } } } Here, addValueChangeListener(\u0026hellip;), the role of the observer at the component level, while the binder synchronises the changes between the input field and the domain object. Without additional infrastructure, this creates a transparent, declarative observation chain from the UI to the model and back to the UI. It is important to note that the button does not trigger a new save, but reads the model state that has already been synchronised by the binder and makes it visible. It thus serves as an action-level observer that confirms and logs the current consistency between UI and model.\n4. Differences between classic Observer and Vaadin Flow # The comparison between the classic Observer Pattern and its implementation in Vaadin Flow reveals that both approaches share the same basic idea, but operate at different levels of abstraction. While the classic pattern relies on purely object-oriented structures – subject and observer communicate directly via defined interfaces – Vaadin Flow shifts observation to a framework-supported environment with integrated event control.\nA significant difference lies in life cycle management. In the classic observer, subjects and observers remain the responsibility of the developer; Registration and deregistration must be done manually, otherwise there is a risk of memory leaks. Vaadin Flow integrates observation into the lifecycle of UI components. Listeners are registered when a component is created and automatically resolved when the component is removed, reducing administrative overhead.\nThreadSecurity also differs significantly. The classic observer pattern works in a local context and has no UI thread binding. Vaadin Flow, on the other hand, must guarantee that UI updates are done exclusively in the UI thread. This forces the use of mechanisms such as UI.access(\u0026hellip;)as soon as events from foreign threads are processed.\nAnother difference concerns the direction and range of the synchronisation. In the classic model, notifications are limited to the objects involved. Vaadin Flow extends this principle to client-server communication: changes to the UI are automatically synchronised to the server, while server-side state changes are, in turn, made visible in the UI on the client side.\nFinally, the granularity of the observation varies. Classic observers usually react to gross changes in state. Vaadin Flow, on the other hand, offers a variety of specialised event types (e.g. ValueChange, Click, Attach/Detach) that allow for very finely tuned reactions. In combination with the binder , a bidirectional coupling between model and UI is created that goes far beyond the original idea of the pattern.\nIn summary, the classic observer pattern provides the theoretical basis. Still, Vaadin Flow abstracts and expands it by integrating lifecycle, thread security, and client-server synchronisation, thereby enabling a higher degree of robustness and productivity.\n5. When does the classic observer pattern still make sense? # Although Vaadin Flow already integrates many observation mechanisms, there are scenarios in which the classic observer pattern can also be used sensibly in modern Vaadin applications. The crucial point here is the separation between UI logic and domain logic. While listeners and binders in Vaadin handle the synchronisation between user input and UI components, events often arise in the business layer that must be managed independently of the UI.\nA typical field of application is internal domain events. For example, a background process can receive a new message, complete a calculation, or import data from an external system. These events should not be directly bound to UI components; instead, they should be distributed throughout the system. Here, the classic observer pattern is suitable for informing interested services or components of such changes without creating a direct link.\nThe pattern also plays a role in the context of integrations with external systems. When a REST service or messaging infrastructure feeds events into the system, the observer pattern can be used to propagate those events within the domain first. Only in the second step is it decided whether and how this information is passed on to a Vaadin UI. This keeps the dependency on the UI low, and the domain layer remains usable outside of a UI context.\nAnother reason for using the classic observer pattern is testability and reusability. Unit testing at the domain layer is easier to perform when observation mechanisms do not depend on Vaadin-specific APIs. This allows complex interactions to be tested in isolation before they are later wired to the UI.\nIn summary, the classic observer pattern complements the mechanisms available in Vaadin Flow for UI-independent, domain-internal events or integrations with external sources. It enables loose coupling, increases testability and ensures clear demarcations between layers – a principle that is a decisive advantage, especially in more complex applications.\n6. Example: Combination of Observer Pattern and Vaadin Flow (with external producer) # An efficient variant arises when messages are not generated by the UI itself, but come from an external producer outside the UI life cycle. Typical sources are background processes, REST integrations or message queues. The Observer Pattern ensures that these events can be distributed to all interested components, regardless of the UI.\nThe following example extends the previous construction: A MessageProducer periodically generates messages in a separate thread and passes them to the MessageService. The Vaadin view remains a pure observer, reflecting only the messages in the UI.\n//Bootstrap: starts the producer on startup final class ApplicationBootstrap { private static final MessageService SERVICE = new MessageService(); static { MessageProducer.start(SERVICE); } private ApplicationBootstrap() { } static MessageService messageService() { return SERVICE; } } //External producer that periodically generates news public class MessageProducer { private static ScheduledExecutorService scheduler; private MessageProducer() { } static void start(MessageService service) { scheduler = Executors.newSingleThreadScheduledExecutor(); scheduler.scheduleAtFixedRate( new CronJob(service), 0, 2, SECONDS); } static void stop() { if (scheduler != null) { scheduler.shutdownNow(); } } public record CronJob(MessageService service) implements Runnable, HasLogger { @Override public void run() { logger().info(\u0026#34;Next message will be sent..\u0026#34;); service.newMessage(\u0026#34;Tick @ \u0026#34; + Instant.now()); } } } //Domain Service public class MessageService implements Subject\u0026lt;String\u0026gt; , HasLogger { private final List\u0026lt;Observer\u0026lt;String\u0026gt;\u0026gt; observers = new CopyOnWriteArrayList\u0026lt;\u0026gt;(); @Override public void addObserver(Observer\u0026lt;String\u0026gt; o) { logger().info(\u0026#34;New Observer added: {}\u0026#34;, o); observers.add(o); } @Override public void removeObserver(Observer\u0026lt;String\u0026gt; o) { logger().info(\u0026#34;Observer removed: {}\u0026#34;, o); observers.remove(o); } @Override public void notifyObservers(String value) { logger().info(\u0026#34;Notifying {} observers with the value {} \u0026#34;, observers.size(), value); observers.forEach(o -\u0026gt; o.onUpdate(value)); } public void newMessage(String msg) { notifyObservers(msg); } } public interface Observer\u0026lt;T\u0026gt; { void onUpdate(T value); } public interface Subject\u0026lt;T\u0026gt; { void addObserver(Observer\u0026lt;T\u0026gt; o); void removeObserver(Observer\u0026lt;T\u0026gt; o); void notifyObservers(T value); } This structure creates a complete message flow producer → domain → UI. The domain layer is independent of Vaadin and can be tested in isolation. The UI remains limited to the display and updates itself exclusively in the UI thread.\nThe new messages appear every two seconds, as the MessageProducer is configured in the background with scheduleAtFixedRate. When it starts, it immediately generates a message and then periodically generates a new one every two seconds. These are distributed to the registered observers via the MessageService. Important: For server-side changes to be reflected in the browser without user interaction, ServerPush or Polling must be enabled. In this example, @Push on the View ensures that ui.access(\u0026hellip;) actively sends the changes to the client. Alternatively, polling (e.g. @Poll(2000) or UI#setPollInterval). The view receives the events and sets the text in the div using UI.access(\u0026hellip;) in UI Thread, so that the most recent message is visible.\nThis uses the Observer Pattern to integrate external events into a Vaadin Flow application reliably.\n7. Best Practices and Pitfalls # The integration of the classic Observer Pattern into a Vaadin Flow application brings both advantages and specific challenges. To ensure a robust and maintainable architecture, several best practices should be considered.\nObserver lifecycle management. A common mistake is to leave Observer permanently registered in the service, even if the associated UI component has already been destroyed. This leads to memory leaks and unexpected calls to views that no longer exist. Therefore, observers in Vaadin views should always be registered inonAttach and removed in onDetach. In this way, the observation remains clearly linked to the life cycle of the UI.\nUse of suitable data structures\nInstead of simple lists, it is recommended to use concurrent, duplicate-free data structures, such as ConcurrentHashMap or CopyOnWriteArraySet, for managing observers. This avoids duplicate registrations and ensures thread safety, even if multiple threads add or remove observers at the same time.\nThread security and UI updates. Since external events are often generated in separate threads, it is imperative to install UI updates in Vaadin viaUI.access(\u0026hellip;) be performed. Otherwise, there is a risk of race conditions and exceptions, because the UI components may only be changed from within the UI thread. Additionally, server push or polling should be enabled to ensure that changes are sent to the client without user interaction.\nDomain and UI demarcation. The domain layer should remain completely free of Vaadin-specific dependencies. This ensures that business logic can be tested and reused outside of the UI context. The UI only registers itself as an observer and takes over the representation of the events delivered by the domain.\nError handling and logging\nEvents should be handled robustly to prevent individual errors from interrupting the entire notification flow. Clean logging (e.g. via a common HasLogger interface) facilitates the analysis and traceability of the event flow.\nSummary. The main pitfalls lie in improper lifecycle management, a lack of thread security, and overly tight coupling between the UI and domain. When these aspects are taken into account, the combination of the Observer Pattern and Vaadin Flow enables a clear separation of responsibilities, testable business logic, and a reactive, user-friendly interface.\n8. Conclusion # A look at the Observer Pattern in combination with Vaadin Flow reveals that it remains a relevant and compelling design pattern, whose benefits extend beyond classic desktop or console applications. In its original form, it provides apparent decoupling of state changes and their processing, thus creating a clean basis for loosely coupled architectures.\nVaadin Flow abstracts and extends this approach by integrating observation mechanisms directly into the lifecycle of UI components, providing a variety of specialised events, and automating synchronisation between server and client. This relieves the developer of many tasks that could still be solved manually in the classic observer pattern, such as memory management or thread security.\nNevertheless, the classic observer pattern remains essential in areas where UI-independent events occur or external systems are connected. The combination of both approaches – a clearly defined domain layer with observers and a Vaadin UI based on it – creates an architecture that is both loosely coupled, testable, and extensible.\nOverall, it can be said that the Observer Pattern forms the theoretical basis, while Vaadin Flow provides the practical implementation for modern web applications. Those who consciously combine both approaches benefit from robust and flexible systems that are prepared for future requirements.\n","date":"1 September 2025","externalUrl":null,"permalink":"/posts/how-and-why-to-use-the-classic-observer-pattern-in-vaadin-flow/","section":"Posts","summary":"1. Introduction and motivation # The observer pattern is one of the basic design patterns of software development and is traditionally used to decouple state changes and process them. Its origins lie in the development of graphical user interfaces, where a shift in the data model required synchronising several views immediately, without a direct link between these views. This pattern quickly established itself as the standard solution to promote loosely coupled architectures.\n","title":"How and why to use the classic Observer pattern in Vaadin Flow","type":"posts"},{"content":"","date":"1 September 2025","externalUrl":null,"permalink":"/tags/observer/","section":"Tags","summary":"","title":"Observer","type":"tags"},{"content":"Password security is an often underestimated but critical topic in software development. Databases containing millions of user logins are repeatedly compromised - and shockingly, often, it turns out that passwords have been stored in plain text. This gives attackers direct access to sensitive account data and opens the door to identity theft, account takeovers and other attacks.\nIn this blog post, we discuss why passwords should always be stored hashed, the attack methods available, and how you can implement an initial secure implementation with Java in your application. We also examine the differences between PBKDF2, BCrypt, and Argon2 and explain best practices for handling passwords in software development.\nPasswords and hashing What are brute force and rainbow table attacks on passwords? And how do I do this now in my application? How safe is PBKDF2? Using stronger hashing algorithms like Argon2 Application of Argon2 in Java Generate the SALT value. What is the procedure for checking the username-password combination during the login process? A few more words about strings in Java So what can you do about it now? Conclusion Passwords and hashing # Passwords should never be stored in plain text but should always be hashed to ensure the security of user data and prevent misuse. If a database is compromised and passwords are only stored in plain text, attackers have direct access to users\u0026rsquo; sensitive credentials. This can have serious consequences, as many people use the same password for multiple services. Hashing passwords significantly reduces this risk because attackers only see the hash values ​​, not the actual ones.\nA key advantage of hashing is its one-way function. A hash value can be generated from a password, but inferring the original password is virtually impossible. This makes it extremely difficult to misuse the access data resulting from a data leak. This protection mechanism applies not only to external attacks but also to internal security risks. If passwords are stored in plain text, employees with access to the database, for example, could view and misuse this information. Hashed passwords largely eliminate such insider threats.\nIn addition, compliance with legal requirements and security standards such as the General Data Protection Regulation (GDPR) or the Payment Card Industry Data Security Standard (PCI-DSS) requires the protection of passwords. Hashing passwords is important to meet these standards and avoid legal consequences.\nWhat are brute force and rainbow table attacks on passwords? # Brute force attacks and rainbow table attacks are two methods attackers use to decrypt passwords or other secrets. A Brute force attack works by systematically trying every possible password combination until finding the right one. We start with the simplest combinations, such as “aaa” or “1234”, and check every possible variant. Although brute force attacks can theoretically always be successful, their effectiveness depends heavily on the complexity of the password and computing power. A short password can be cracked in a few seconds, while a long and complex password increases the effort significantly. Measures such as longer passwords, limiting the number of login attempts per unit of time (e.g. blocking after several failed attempts) and the use of hashing algorithms with many iterations make brute force attacks significantly more difficult and time-consuming.\nIn contrast, the Rainbow table attacks pre-built tables that contain many hash values ​​and their associated plaintext passwords. Attackers compare the stored hash of a password with the hashes in the table to find the original password. This method is significantly faster than brute force because the passwords must not be re-hashed every time. However, rainbow tables only work if the hashing method does not use a salt value. A salt value is a random value added to each password hash, so even identical passwords produce different hashes. Without salt, attackers could use the same table across many other systems, but this is no longer possible with salt.\nUsing modern hashing algorithms such as PBKDF2, BCrypt, or Argon2 is crucial to protecting yourself from both attack methods. These algorithms combine salts and multiple iterations to significantly increase attacker effort. Additionally, long and complex passwords make brute-force attacks virtually impossible as the number of possible combinations increases exponentially. In summary, combining strong passwords, salts, and secure hashing algorithms effectively defends against brute force and rainbow table attacks.\nAnd how do I do this now in my application? # To securely hash a password in Java, it is recommended to use a specialised hash function like PBKDF2 , BCrypt , or Argon2. These algorithms are specifically designed to make attacks such as brute force or rainbow table attacks more difficult. One way to implement this with the Java standard library is to use PBKDF2.\nFirst, a Salt is generated, a random sequence of bytes generated for each password individually to ensure that two identical passwords have different hashes. The class SecureRandom can be used to create a 16-byte salt. Then, with the class PBEKeySpec, a key is generated based on the password, the salt, the desired number of iterations (e.g. 65,536) and the key length (e.g. 256 bits). With the help of SecretKeyFactory and the algorithm specification \u0026ldquo;PBKDF2WithHmacSHA256\u0026rdquo; the password hash is created. The resulting hash is finally encoded in Base64 to store it in a readable form.\nThe hashed password is often stored along with the salt, separated by a colon (:). This makes it easier to verify later by extracting the salt again and using it for the hash calculation. In addition, there are also external libraries such as Spring Security or BouncyCastle , which allow easier integration of algorithms like BCrypt or Argon2 make possible. These often offer even more security and flexibility.\nimport java.security.NoSuchAlgorithmException; import java.security.spec.InvalidKeySpecException; import java.security.spec.KeySpec; import java.util.Base64; import javax.crypto.SecretKeyFactory; import javax.crypto.spec.PBEKeySpec; public class PasswordHasher { private static final int ITERATIONS = 65536; private static final int KEY_LENGTH = 256; private static final String ALGORITHM = \u0026#34;PBKDF2WithHmacSHA256\u0026#34;; public static String hashPassword(String password, String salt) { try { KeySpec spec = new PBEKeySpec( password.toCharArray(), salt.getBytes(), ITERATIONS, KEY_LENGTH); SecretKeyFactory factory = SecretKeyFactory.getInstance(ALGORITHM); byte[] hash = factory.generateSecret(spec).getEncoded(); return Base64.getEncoder().encodeToString(hash); } catch (NoSuchAlgorithmException | InvalidKeySpecException e) { throw new RuntimeException(\u0026#34;Error while hashing password\u0026#34;, e); } } public static void main(String[] args) { String password = \u0026#34;myPassword\u0026#34;; String salt = \u0026#34;randomSalt\u0026#34;; String hashedPassword = hashPassword(password, salt); System.out.println(\u0026#34;hasehd password: \u0026#34; + hashedPassword); } } However, there are a few points here that you should take a closer look at.\nHow safe is PBKDF2? # PBKDF2 is a proven and secure password hashing algorithm, but its security depends heavily on the correct implementation and parameters used. Various factors influence the safety of PBKDF2. A key aspect is the number of iterations, representing the work factor. The higher the number of iterations, the more computing power is required for hashing. Experts recommend at least 100,000 iterations, although even higher values ​​are often chosen for modern applications to make brute-force attacks more difficult. Another crucial factor is the salt value, a random and unique value that is regenerated for each password. In addition, PBKDF2 can be configured with a variable key length, for example, 256 bits, which increases security. Modern applications often use hash functions such as SHA-256 or SHA-512.\nPBKDF2 has several strengths. The algorithm is proven and has been used for years in security-critical applications such as WPA2 and encrypted storage systems. It is easy to implement as extensive support in many programming languages ​​and libraries is based on a recognised standard defined in RFC 8018. Nevertheless, PBKDF2 also has weaknesses. The algorithm is not optimised explicitly against GPU or ASIC-based attacks, allowing attackers with specialised hardware to efficiently carry out brute force attacks. Compared to modern algorithms such as Argon2, PBKDF2 requires a higher number of iterations to achieve similar levels of security. In addition, the further development of PBKDF2 is progressing more slowly, which means it is considered less adapted to current threats than modern alternatives such as Argon2.\nHowever, PBKDF2 can be used safely if a few key conditions are met: a unique salt value, the iteration count should be at least 100,000 (ideally more), and a modern hash function such as SHA-256 should be used. In addition, strong password guidelines, such as a sufficient minimum length, should complement security.\nArgon2 is increasingly recommended as an alternative to PBKDF2. Argon2, the Password Hashing Competition (PHC) winner in 2015, is more modern and better adapted to current threats. Its memory intensity provides better protection against GPU and ASIC attacks and offers flexible configuration options regarding work factor, memory requirements and parallelism.\nIn summary, PBKDF2 is secure when implemented correctly but has weaknesses compared to specialised hardware. Argon2 is preferable for new applications because it is better suited to modern security requirements.\nUsing stronger hashing algorithms like Argon2 # Argon2 is a modern algorithm for secure password hashing and was developed in 2015 as part of the Password Hashing Competition (PHC). It is considered one of the most secure approaches to storing passwords today and, as already mentioned, offers adequate protection against brute force attacks as well as attacks using GPUs or specialised hardware solutions such as ASICs. This is achieved due to the fact that Argon2 is both memory and compute-intensive, forcing attackers with parallel hardware to expend significant resources.\nArgon2 exists in three variants, each optimised for different use cases. The first variant, Argon2i , is designed to be particularly memory-safe. It performs memory-intensive operations independently of the input data and is resistant to side-channel attacks such as timing attacks. This makes Argon2i ideal for applications where data protection is a top priority.\nThe second variant, Argon2d , is specifically optimised for attack resistance. It works data-dependently, which makes it particularly robust against GPU-based attacks. However, Argon2d is more vulnerable to side-channel attacks and is, therefore, less suitable for securing sensitive data.\nThe third variant, Argon2id, offers a balanced combination of both approaches. It starts with a memory-safe approach, like that used by Argon2i, and then moves to a data-dependent process that ensures attack resistance. This mix makes Argon2id the preferred choice for most use cases, as it combines data protection and attack resilience.\nOne of Argon2\u0026rsquo;s key strengths is its customizability. The algorithm allows developers to configure three primary parameters: memory consumption, computational cost and parallelism. Memory consumption defines how much memory is used during the hashing operation and makes parallel hardware attacks expensive. Computational cost indicates the number of iterations the algorithm goes through, while parallelism determines the number of threads working simultaneously. By adjusting these parameters, the algorithm can be tailored to the specific requirements of an application or the available hardware.\nAnother advantage of Argon2 is its resistance to modern attacks. The combination of memory and computing-intensive processes makes it difficult for attackers to crack passwords using brute force or specialised hardware. This makes Argon2 ideal for use in safety-critical applications.\nArgon2 is used in many modern cryptography and password management libraries. Developers can implement the algorithm in various programming languages ​​, such as Java, Python, or C.\nIn summary, Argon2 is one of the safest password-hashing algorithms. The variant Argon2id is especially recommended as a standard for new projects because it provides protection against both side-channel attacks and high attack resistance.\nApplication of Argon2 in Java # One Argon2 in Java To use it, you can use libraries like Jargon2 or Bouncy Castle because Java does not natively support Argon2. With Jargon2 , Argon2 is very easy to integrate and use. To do this, first, add the following Maven dependency:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.kosprov\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;jargon2-api\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;Latest Version Number\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; The code to create a hash and verify a password might look like this:\nimport com.kosprov.jargon2.api.Jargon2; public class Argon2Example { public static void main(String[] args) { String password = \u0026#34;DeinSicheresPasswort\u0026#34;; Jargon2.Hasher hasher = Jargon2.jargon2Hasher() .type(Jargon2.Type.ARGON2id) .memoryCost(65536) .timeCost(3) .parallelism(4) .saltLength(16) .hashLength(32); String hashedPassword = hasher .password(password.getBytes()) .encodedHash(); System.out.println(\u0026#34;hashed password: \u0026#34; + hashedPassword); boolean matches = Jargon2.jargon2Verifier() .hash(hashedPassword) .password(password.getBytes()) .verifyEncoded(); System.out.println(\u0026#34;password correct: \u0026#34; + matches); } } If you want to use a more comprehensive cryptography library, you can use Bouncy Castle. The corresponding Maven dependency is:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.bouncycastle\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;bcprov-jdk18on\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;Latest Version Number\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; An example of using Argon2 with Bouncy Castle looks like this:\nimport org.bouncycastle.crypto.params.Argon2Parameters; import org.bouncycastle.crypto.generators.Argon2BytesGenerator; import java.security.SecureRandom; public class Argon2BouncyCastleExample { public static void main(String[] args) { String password = \u0026#34;securePassword\u0026#34;; byte[] salt = new byte[16]; new SecureRandom().nextBytes(salt); Argon2Parameters.Builder builder = new Argon2Parameters .Builder(Argon2Parameters.ARGON2_id) .withSalt(salt) .withMemoryPowOfTwo(16) .withParallelism(4) .withIterations(3); Argon2BytesGenerator generator = new Argon2BytesGenerator(); generator.init(builder.build()); byte[] hash = new byte[32]; generator.generateBytes(password.getBytes(), hash); System.out.println(\u0026#34;hashed password: \u0026#34; + bytesToHex(hash)); } private static String bytesToHex(byte[] bytes) { StringBuilder hexString = new StringBuilder(); for (byte b : bytes) { hexString.append(String.format(\u0026#34;%02x\u0026#34;, b)); } return hexString.toString(); } } Jargon2 was developed specifically for Argon2 and is, therefore, easy to use Bouncy Castle for more complex cryptographic requirements. It is recommended in most cases for the pure use of Argon2.\nGenerate the SALT value. # So far, it has always been mentioned that producing a good SALT value is important. But how can you do that?\nWe will examine the topic in detail in the next blog post. Unfortunately, at this point, it would go beyond the scope of this article.\nIn this blog post, we will examine a basic initial implementation. You will quickly be confronted with an implementation that could look like this.\npublic String generateSalt(int length) { SecureRandom random = new SecureRandom(); byte[] salt = new byte[length]; random.nextBytes(salt); return Base64.getEncoder().encodeToString(salt); } However, there are some minor comments here.\nBasically, creating a salt with a newly instantiated SecureRandom is not fundamentally wrong in any method, but it increases the chance that the same seeds are used at very short intervals or under certain circumstances. In practice, that is rarely a problem; however, it is good practice to have a single (static) instance of SecureRandom to be used per application (or per class).\nWhy?\nSecureRandom gets its seed (among other things) from the operating system (e.g /dev/urandom in Linux). Each re-creation of one SecureRandom can lead to unnecessary system load and, theoretically, minimal increased risk of repetitions. At just one SecureRandominstance, a pseudo-random number generator is continued internally, making duplicates very unlikely. import java.security.SecureRandom; import java.util.Base64; public class SaltGenerator { private static final SecureRandom RANDOM = new SecureRandom(); public static String generateSalt(int length) { byte[] salt = new byte[length]; RANDOM.nextBytes(salt); return Base64.getEncoder().encodeToString(salt); } } This way, (a) is not repeated every time it is called SecureRandom, and (b) reduces the risk of randomly generating identical salts. Of course, purely statistically speaking, collisions can theoretically still occur, but the probability is negligible if the salt length is long enough.\nHowever, this is only the beginning of the discussion; more on that in the second part.\nWhat is the procedure for checking the username-password combination during the login process? # A standardised procedure is followed to check the combination of user name and password during a login process, ensuring that the data is processed securely. First, the user enters their login details via a form transmitted over HTTPS. The server then looks up the username in the database to retrieve relevant information such as password hash and salt. Care should be taken not to reveal whether a user exists.\nThe stored password hash is then compared with a recalculated hash of the entered password using the stored salt. If the values ​​match, the login is considered successful; otherwise, a generic error message is returned so as not to reveal additional information.\nAfter successful verification, a session or JSON Web Token (JWT) is created to maintain the user\u0026rsquo;s authentication. The token does not contain any sensitive information, such as passwords.\nA few more words about strings in Java # If we imagine that a password is in a text field of a (web) application, we will receive this password as a string. Yes, some will say, but what could be bad about that?\nTo do this, we need to examine briefly how strings are handled in the JVM and where attack vectors can lie.\nIn Java, strings are immutable, which means that once created, String-Object can no longer be changed. When a string is manipulated through concatenation, a new string object is created in memory while the old one continues to exist until it is removed by the garbage collector (GC). This behaviour can be problematic when sensitive data, such as passwords or cryptographic keys, is stored in strings. Because they cannot be overwritten directly, they may remain in memory longer than necessary and are potentially visible in a memory dump or readable by an attacker. Another problem is that the developer has no control over when the garbage collector removes the sensitive data. This can cause such data to remain in memory for an extended period and possibly become visible in logs or debugging tools.\nSo what can you do about it now? # A safer approach is to use this instead of strings char[] because they are changeable, and the memory can be specifically overwritten. This helps minimise the retention time of sensitive data, especially in security-critical applications where memory dumps or debugging tools could provide memory access.\nThe advantage of char[] lies in direct memory management: While the garbage collector decides itself when to remove objects, a char[]-Array is explicitly overwritten and thus immediately deleted. This significantly reduces the risk of unauthorised access.\nimport java.util.Arrays; public class SensitiveDataExample { public static void main(String[] args) { char[] password = {\u0026#39;s\u0026#39;, \u0026#39;e\u0026#39;, \u0026#39;c\u0026#39;, \u0026#39;r\u0026#39;, \u0026#39;e\u0026#39;, \u0026#39;t\u0026#39;}; try { processPassword(password); } finally { // Überschreiben des Speichers mit Dummy-Daten Arrays.fill(password, \u0026#39;\\0\u0026#39;); } } private static void processPassword(char[] password) { // Beispielhafte Verarbeitung System.out.println(\u0026#34;Passwort verarbeitet: \u0026#34; + String.valueOf(password)); } } Next to char[], additional security measures should be taken, such as encrypting sensitive data and using SecretKeySpec from the package javax.crypto.spec. for cryptographic keys. This class allows handling cryptographic keys in byte arrays that can be overwritten after use, but I will write a separate blog post about that…\nConclusion # Secure storage of passwords is an essential part of every application to protect user data from misuse and unauthorised access. Plain text passwords pose a significant risk and should never be saved directly. Instead, modern hashing algorithms such as PBKDF2, BCrypt or Argon2 must be used to ensure security.\nBrute force and rainbow table attacks illustrate the importance of salts and sufficiently complex hashing mechanisms. The risk of such attacks can be significantly reduced by using random salts and iterative hashing methods. Argon2id, in particular, offers high resistance to modern attack methods due to its storage and computing intensity and is recommended as the preferred solution.\nIn addition, passwords within the application should also be processed carefully. Using char[] instead of String to store sensitive data can help prevent unwanted memory leaks. It is also important not to store passwords unsecured in memory or in logs.\nSecure password hashing procedures are a technical best practice and a legal requirement in many areas.\n","date":"29 August 2025","externalUrl":null,"permalink":"/posts/password-security-why-hashing-is-essential/","section":"Posts","summary":"Password security is an often underestimated but critical topic in software development. Databases containing millions of user logins are repeatedly compromised - and shockingly, often, it turns out that passwords have been stored in plain text. This gives attackers direct access to sensitive account data and opens the door to identity theft, account takeovers and other attacks.\n","title":"Password Security: Why Hashing is Essential","type":"posts"},{"content":"","date":"29 August 2025","externalUrl":null,"permalink":"/categories/secure-coding-practices/","section":"Categories","summary":"","title":"Secure Coding Practices","type":"categories"},{"content":"","date":"29 August 2025","externalUrl":null,"permalink":"/tags/secure-coding-practices/","section":"Tags","summary":"","title":"Secure Coding Practices","type":"tags"},{"content":"","date":"29 August 2025","externalUrl":null,"permalink":"/categories/security/","section":"Categories","summary":"","title":"Security","type":"categories"},{"content":"","date":"29 August 2025","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"","date":"19 August 2025","externalUrl":null,"permalink":"/tags/flow/","section":"Tags","summary":"","title":"Flow","type":"tags"},{"content":"From my experience, Vaadin has always stood out from other Java frameworks. Of course, it enables the creation of modern web UIs, but the real difference lies in its component architecture. This is not conceived as a short-term aid, but is consistently designed for maintainability and flexibility. It creates the possibility of running applications stably for many years while still being able to extend them step by step.\nWhether I use a component from the official library, develop my own, or integrate an npm package, Vaadin ensures that all building blocks follow a clear pattern. The connection between server and client remains consistent and reliable. This has allowed me to further develop applications over time without ever being forced to rebuild them entirely from scratch.\nNote: This post builds on and reflects further upon the excellent original article by Sami Ekblad, who provided a concise overview of what makes Vaadin components special. My intention here is to expand on those ideas with reflections from my own long-term use in practice. -https://dev.to/samiekblad/what-makes-vaadin-components-special-9oo-temp-slug-1862339\nMore Than Just UI Widgets\nIn my practical work, I have observed that libraries such as React or Lit understand their components primarily as frontend building blocks. Vaadin deliberately takes a different approach. For me, the focus is on longevity and extensibility , which are crucial in long-term projects. I have seen applications operated with Vaadin for more than a decade, and the underlying component architecture is designed precisely for this.\nWhat I find particularly interesting is the possibility of implementing UI components either entirely server-side, purely client-side, or in a hybrid form. This freedom of choice is the real strength that sets Vaadin apart from traditional UI libraries and gives me the flexibility to choose the most appropriate solution for each use case.\nThree Approaches to Vaadin Components\n1. Server-Side Components\nThis is the most traditional route: development is done exclusively in Java. HTML templates or JavaScript are not necessary. With existing building blocks such as TextField, Button, or Grid, entire interfaces can be assembled. These can, in turn, evolve into higher-level components, such as an InvoiceEditor that directly integrates business logic and input validation.\nStrengths:\nServer-side components offer maximum control over the validation and security of user input, as all checks take place directly on the server and are thus protected against manipulation. At the same time, there is no dependency on frontend technologies, which reduces complexity and simplifies maintenance. Another advantage is the possibility of generating user interfaces dynamically at runtime, allowing flexible responses to changes in business logic or context. In addition, existing Java libraries and enterprise logic can be integrated seamlessly, creating a smooth transition between UI and backend.\nWeaknesses:\nEvery user interaction requires a request to the server, which in scenarios with very high interactivity can lead to noticeable delays. In particular, with animations or complex drag-and-drop mechanisms, this approach reaches its limits, as such functions demand an immediate response in the browser that cannot be achieved without client-side logic.\n2. Client-Server Components\nHere, client-side web components (Lit, TypeScript, or Vanilla) are combined with a Vaadin Java API. This architecture unites a rich user experience with secure backend validation.\nExample: A date picker with keyboard shortcuts, colour transitions and smooth animations, while still ensuring that inputs are reliably validated on the server.\nStrengths:\nClient-server components provide complete control over the DOM and all interactions in the browser. This makes it possible to realise complex and responsive user interfaces that keep pace with modern web standards. At the same time, business logic does not remain unprotected in the frontend but is validated and executed server-side in Java, ensuring a high level of security. Another benefit is the clear separation between UI and server logic, since communication is handled in a structured way by the framework, promoting clarity and maintainability.\nWeaknesses:\nA disadvantage is that developers need solid knowledge of both Java and TypeScript. This dual requirement can raise the entry barrier and lengthen the learning curve, since one must be familiar not only with server-side concepts but also with the particularities of client-side development. Especially in teams that have so far focused predominantly on one language, this can demand additional training and a more intensive onboarding process.\n3. Integration of Third-Party Components\nThe modern web is rich in high-quality custom elements available via npm. With Vaadin, these elements can be easily integrated into an application. A small amount of TypeScript is sufficient to build the bridge to the Java world.\nStrengths:\nThe integration of existing web components is particularly fast and straightforward, since ready-made elements from the npm ecosystem can be included with minimal effort. Developers benefit from the fact that only a small amount of boilerplate code is required to make a component usable. This leaves more scope for actual application logic. These integrated elements are also well-suited for visual enhancements or additional convenience features that may not be business-critical but still improve the user experience noticeably.\nWeaknesses:\nOne drawback of integrating third-party components is that server-side validation is not automatically provided and must instead be implemented individually. Developers, therefore, need to add backend logic to ensure that inputs are reliably checked and possible manipulation is prevented. Another point concerns visual design: not all external components integrate seamlessly with the Vaadin design system. This can result in visual inconsistencies that impair a uniform appearance of the application and may require additional adaptation effort.\nMaking the Right Choice\nThe real question is not to commit permanently to a single approach. Each of the described variants brings specific strengths and weaknesses, which can be relevant in different contexts. In my projects, I have found the most outstanding value when combining the approaches. For example, I have been able to use server-side forms with clear validation logic alongside highly interactive client widgets, while also incorporating external components from the npm ecosystem. In this way, an application emerges in which different building blocks coexist harmoniously without losing their semantic meaning. At its core, a button remains a button – regardless of whether it is implemented server-side, client-side, or in a hybrid form.\nConclusion: Flexibility as a Guiding Principle\nIn my experience, Vaadin’s true strength does not lie solely in its library or the choice between Java and TypeScript, but in the flexibility that allows me as a developer to place logic exactly where it delivers the most significant benefit in terms of performance, security and maintainability. Particularly useful is the fact that existing components can be combined seamlessly with new technical approaches, enabling applications to grow continuously without requiring a fundamental redevelopment. This possibility of evolving systems organically has proved to be a decisive advantage in long-term projects.\nFor me, Vaadin components are therefore far more than simple UI building blocks. They embody an architectural principle designed for sustainability and future viability. This creates a stable foundation on which modern user interfaces can be developed that adapt flexibly to new requirements. In my work, I have repeatedly found that this approach allows new functionality to be introduced step by step without risking breaks in the existing architecture or being forced to restart a project from scratch.\nIn conclusion, what have your experiences been with the different approaches? Have you already worked with hybrid components that combine server-side and client-side logic, or have you instead found situations where a consistently pursued style has brought advantages? I am particularly interested in how you use these possibilities in practice, what challenges you have encountered, and which patterns have proved helpful for you.\n","date":"19 August 2025","externalUrl":null,"permalink":"/posts/what-makes-vaadin-components-special/","section":"Posts","summary":"From my experience, Vaadin has always stood out from other Java frameworks. Of course, it enables the creation of modern web UIs, but the real difference lies in its component architecture. This is not conceived as a short-term aid, but is consistently designed for maintainability and flexibility. It creates the possibility of running applications stably for many years while still being able to extend them step by step.\n","title":"What makes Vaadin components special?","type":"posts"},{"content":" 1. Introduction and objectives # The first two parts of this series established the theoretical and practical foundations of a URL shortener in pure Java. We discussed the semantic classification of short URLs, the architecture of a robust mapping system, and the implementation of a REST-based service based on the JDK HTTP server. These efforts resulted in a functional, modularly extensible backend that creates, manages, and efficiently resolves short links. However, a crucial component was missing-a visual interface for direct user interaction with the system. This interface is essential for tasks such as manual link creation, viewing existing mappings, and analysing individual redirects.\n1. Introduction and objectives 2. UI strategy and technology choice 2.1 Why Vaadin Flow for a server-side interface? 2.2 UI paradigms for administration systems 2.3 Differentiation from client-side frameworks 3. User interface architecture 3.1 Integration into the existing WAR structure 3.2 Separation of routing, views and services 3.3 Communication with the core module (DI without frameworks) 4. User guidance and UX design 4.1 Navigation concept: Dashboard, Create, Overview 4.2 Input validation and feedback 4.3 Readability, design system and accessibility 5. Implementation of the UI components with Vaadin Flow (including source code) 5.1 MainLayout: Navigation frame and routing 5.2 CreateView: Input form with validation 5.3 OverviewView: Grid-based management view 5.4 Consistent behaviour for dialogues and notifications (with code) Success stories via Notification Error messages through field-based validation withBinder Destructive actions only after explicit confirmation Interaction and reuse 7. Deploying the UI in the WAR context 7.1 Servlet configuration for Vaadin inweb.xml 7.2 Theme-Integration ohne Frontend-Toolchain 7.3 Operation in Tomcat or Jetty 8. Conclusion and outlook This is precisely where the third part of this series comes in. The focus is on the development of a graphical user interface based on Vaadin Flow. The goal is to provide a web interface developed entirely in Java that eliminates the need for client-side frameworks and integrates seamlessly into the application\u0026rsquo;s existing WAR structure. The decision to use Vaadin Flow follows a pragmatic principle: Anyone who already writes Java should also be able to define interfaces in Java, with a type-safe layout, a declarative component model, and a clear separation between the UI and domain layers.\nUI integration in our project serves not only to enhance functionality but also to achieve a didactic objective. In security-critical applications like URL shorteners, human interaction is key. A well-designed UI not only facilitates convenient operation but also prevents common errors and misuse through appropriate validation, clear feedback, and restrictive permissions concepts. This makes the user interface itself a security-relevant part of the architecture.\nIn the remainder of this article, we will examine the structure, design, and concrete implementation of the Vaadin interface in detail. This ranges from the module architecture to navigation and view components, including advanced features such as QR code generation and expiration time display. The goal remains to implement all functionality exclusively using Java\u0026rsquo;s built-in tools—in the spirit of a transparent, maintainable, and extensible system design.\nThe UI is not just an add-on, but a central component of a comprehensive shortener solution. It caters to both productive use and teaching and exploration, providing a sense of security and reliability.\n2. UI strategy and technology choice # Choosing the right technology for the user interface of a technical service is far more than a matter of taste – it touches on key aspects such as maintainability, security model, development efficiency, and deployment strategy. In the context of our URL shortener, which deliberately avoids external frameworks and was implemented entirely in Java 24, it made sense to choose a solution for the UI that fits seamlessly into this philosophy. The decision was therefore made to use Vaadin Flow , a server-side UI framework powered and developed entirely in Java.\n2.1 Why Vaadin Flow for a server-side interface? # Vaadin Flow enables the development of component-based web applications in Java, without the need for JavaScript, front-end build processes, or manual maintenance of HTML, CSS, or TypeScript artefacts. All UI components, from simple input fields to complex tables, are described declaratively in Java, allowing the UI to seamlessly integrate into the existing modular structure. For a purely Java-based backend logic like our URL shortener, this results in a coherent overall architecture that eliminates the need for language or context switching, instilling confidence in its efficiency.\nThe logic is executed entirely server-side: User interactions trigger events that are processed on the server, with the presentation in the browser synchronised via the Vaadin client. This architecture not only simplifies state management but also reduces potential attack surfaces, as no server-side business logic migrates to the frontend.\n2.2 UI paradigms for administration systems # When designing an admin interface, it\u0026rsquo;s not about aesthetic brilliance, but rather functional clarity, robust interactions, and data accountability. Users of a URL shortener—whether self-hosted, in educational institutions, or within an organisation—need an easy way to manually shorten new URLs, browse existing mappings, check expiration times, and delete potentially critical entries. These functions must be visible, traceable, and cancelable at any time.\nVaadin Flow offers a variety of ready-made components, such as Grid , FormLayout , Dialogue , Notification or Binder that enable efficient implementation of such use cases – including validation, event control, and dynamic data binding. This not only saves development time but also enables a consistent user experience that scales elegantly with increasing complexity.\n2.3 Differentiation from client-side frameworks # Unlike single-page application frameworks such as Angular, React or Vue, Vaadin Flow requires no separate frontend build pipeline , no TypeScript toolchains, and no JSON serialisation of server communication. The entire application, including the UI, can be deployed as a classic WAR Package and run in a servlet container such as Jetty or Tomcat. This not only reduces DevOps effort but also the risk of version conflicts, CDN outages, or cross-origin issues.\nThis server-side approach is particularly suitable for applications where Data sovereignty, security and long-term maintainability are in the foreground – for example, with internal tools, admin consoles, or security-critical services. The integration of session-based security mechanisms, IP filters, or role-based access is also much more controlled in a Vaadin servlet than with headless backends with an external JavaScript frontend.\nThe choice of Vaadin Flow is therefore not just a technological decision, but an expression of an architectural style: stable, server-centric, Java-enabled – ideal for secure, maintainable and explainable systems.\n3. User interface architecture # The user interface is not an isolated component, but an integral part of the system. It must fit seamlessly into the existing module architecture without enforcing domain-specific dependencies or abandoning the loose coupling between core logic and application layers. In a purely Java-based environment without frameworks such as Spring or Jakarta EE, this separation is particularly important because it allows the UI to be viewed as a separate layer, both technically and semantically.\n3.1 Integration into the existing WAR structure # The application is already organised as a multi-tiered Maven project, which is consolidated into a central WAR module. This WAR module knows the servlet context, initialises the HTTP server (for REST) and the VaadinServlet (for the UI), and thus takes on the role of Integration and delivery unit. The previous modules shortener-core(domain) and shortener-api(HTTP routing) remain unaffected.\nThe new UI module –shortener-ui-required – is implemented as an additional Maven module, which is used exclusively by shortener-core. It contains all Vaadin-specific logic, UI components, and views. In the WAR module, this module is then connected via a servlet registration – typically via a Servlet, which is defined in the deployment descriptor (web.xml) or via ServletContainerInitializer- and is integrated.\nThe structure follows the principle: Only the WAR module knows the specific deployment topology. All other modules remain generic and independent.\n3.2 Separation of routing, views and services # Within the UI module, responsibilities are divided:\nRouting and Navigation are centrally controlled via a MainLayout class and with @Route-annotated views. These define the paths within the UI (e.g./, /create, /admin) and automatically take over session and history management from Vaadin.\nViews (approximately CreateView, OverviewView, StatsView) form the interactive interfaces. They integrate Vaadin components such as TextField, Grid, Button, Dialog and provide user interaction. Each view is stateless and delegates to services.\nServices encapsulate the interaction with the business logic. These are not typical Spring beans, but regular Java classes that can be implemented either directly or via the Singleton pattern (ServiceRegistry, StaticFactory), and are instantiated. They call methods shortener-core and take care of conversion, validation and error handling.\nThis structure allows for a clear separation between presentation, application logic, and business logic. It is explicitly based on classic UI architectural patterns such as Model-View-Service (MVS), without the overhead or magic of full-fledged frameworks.\n3.3 Communication with the core module (DI without frameworks) # Since dependency injection containers are not used, the views are deployed with service instances manually or via a central deployment class. A typical approach is to implement a UiServiceLocator , which contains all the required components (e.g.UrlMappingStore, ShortCodeGenerator), is created once and then made available as a singleton.\nFor example, the constructor of a view can look like this:\npublic class OverviewView extends VerticalLayout { private final UrlMappingStore store = UiServiceLocator.getUrlMappingStore(); public OverviewView() { // Build components, display data, define actions } } This form of explicit dependency creation has the advantage that the origin of all components remains traceable, no hidden initialisation takes place, and the entire application context remains transparently controllable – a property that is particularly important in safety-critical systems.\n4. User guidance and UX design # An administration interface is not a classic front end for end users, but a specialised tool – comparable to a precision instrument. Its user interface must be efficient, robust, and explainable. In contrast to dynamic consumer UIs, which boast animations, branding, or complex interactions, clarity, predictability, and freedom from redundancy are paramount here. The design of a Vaadin interface for a URL shortener, therefore, follows a minimalist, function-centric UX strategy.\n4.1 Navigation concept: Dashboard, Create, Overview # From the user’s perspective, the application is divided into three central task areas:\nCreating new short links : A simple input form with target URL validation and an optional custom alias. Speed and clarity are paramount. Overview of existing mappings : A tabular display of all generated shortlinks, supplemented with sorting and filtering options. This allows for targeted searches by alias, target address, or expiration date. Administrative Dashboard : An area for viewing metrics (e.g. number of mappings, frequently used short links) as well as for targeted deletion or extension of entries. Navigation is done via a Vaadin layout with a main menu (e.g. AppLayout or DrawerToggle) and named routes. All routes are directly accessible via URL, with bookmarks and refreshes always leading to a consistent state—an inherent advantage of Vaadin Flow due to server-side session management and declarative routing.\n4.2 Input validation and feedback # A key UX criterion is the correctness of the inputs. When creating a new mapping, it must be ensured that\nThe input field for the URL is not empty, The URL is syntactically valid (including http/https), optional aliases are not already assigned or reserved. These validations are performed immediately upon input (client-side via Binder) and again server-side in the service layer, whereby the view displays corresponding error messages as Notification , ErrorMessage or directly in the field context. A deliberate UX detail: In the event of errors, the focus remains on the field, the previous input is not deleted, and a visual highlight appears, instead of a general error dialogue.\nSuccessful processes (e.g., successfully created a short link) lead to clear success messages, including immediate copying options, e.g., via a button with an icon (\u0026ldquo;copy to clipboard\u0026rdquo;) to support cross-media use.\n4.3 Readability, design system and accessibility # The UI is functional, not decorative. Accordingly, attention is paid to a clear visual grid: sufficient white space, consistent labels, understandable icons, and no unnecessary animations. Colours serve solely for semantic marking (e.g., red for deletion actions, green for success messages), not for stylistic individualisation.\nVaadin Flow comes with a solid standard design that extends @Theme, or custom CSS classes can be extended, but only where functionally necessary. The entire application remains fully keyboard-operated, screen reader-compatible, and high-contrast enough for low-barrier use. This is especially important in contexts where administrators work in security-critical areas with increased accessibility requirements.\n5. Implementation of the UI components with Vaadin Flow (including source code) # 5.1 MainLayout: Navigation frame and routing # The MainLayout forms the common structure of all views. It is based on the AppLayout and defines a vertical navigation menu with links to the central views:\n@Theme(\u0026#34;shortener-theme\u0026#34;) public class MainLayout extends AppLayout { public MainLayout() { DrawerToggle toggle = new DrawerToggle(); H1 title = new H1(\u0026#34;URL Shortener Admin\u0026#34;); title.getStyle().set(\u0026#34;font-size\u0026#34;, \u0026#34;var(--lumo-font-size-l)\u0026#34;) .set(\u0026#34;margin\u0026#34;, \u0026#34;0\u0026#34;); HorizontalLayout header = new HorizontalLayout(toggle, title); header.setDefaultVerticalComponentAlignment(Alignment.CENTER); header.setWidthFull(); header.setPadding(true); addToNavbar(header); RouterLink createLink = new RouterLink(\u0026#34;Erstellen\u0026#34;, CreateView.class); RouterLink overviewLink = new RouterLink(\u0026#34;Übersicht\u0026#34;, OverviewView.class); VerticalLayout menu = new VerticalLayout(createLink, overviewLink); menu.setPadding(false); menu.setSpacing(false); menu.setSizeFull(); addToDrawer(menu); } } This structure separates navigation logic from application logic. New views only need to be @Route(\u0026hellip;, layout = MainLayout.class) and connected.\n5.2 CreateView: Input form with validation # The view for creating new short links consists of a TextField for the target URL, an optional alias field, and a button for generating the mapping. Simple validation logic is provided via the binder:\n@Route(value = \u0026#34;\u0026#34;, layout = MainLayout.class) public class CreateView extends VerticalLayout { private final UrlMappingStore store = UiServiceLocator.getUrlMappingStore(); public CreateView() { setSpacing(true); setPadding(true); TextField urlField = new TextField(\u0026#34;Target URL\u0026#34;); urlField.setWidthFull(); TextField aliasField = new TextField(\u0026#34;Alias (optional)\u0026#34;); aliasField.setWidth(\u0026#34;300px\u0026#34;); Button shortenButton = new Button(\u0026#34;Shorten\u0026#34;); shortenButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY); HorizontalLayout actions = new HorizontalLayout(shortenButton); actions.setAlignItems(Alignment.END); Binder\u0026lt;ShortenRequest\u0026gt; binder = new Binder\u0026lt;\u0026gt;(ShortenRequest.class); ShortenRequest request = new ShortenRequest(); binder.forField(urlField) .asRequired(\u0026#34;URL must not be empty\u0026#34;) .withValidator(url -\u0026gt; url.startsWith(\u0026#34;http://\u0026#34;) || url.startsWith(\u0026#34;https://\u0026#34;), \u0026#34;Only HTTP(S) URLs allowed\u0026#34;) .bind(ShortenRequest::url, ShortenRequest::url); binder.forField(aliasField) .bind(ShortenRequest::alias, ShortenRequest::alias); shortenButton.addClickListener(click -\u0026gt; { if (binder.writeBeanIfValid(request)) { Optional\u0026lt;String\u0026gt; code = createShortCode(request); code.ifPresentOrElse(c -\u0026gt; { Notification.show(\u0026#34;Short link created: \u0026#34; + c); urlField.clear(); aliasField.clear(); }, () -\u0026gt; Notification.show(\u0026#34;Alias already assigned or error saving\u0026#34;, 3000, Position.MIDDLE)); } }); add(new H2(\u0026#34;Create new short link\u0026#34;), urlField, aliasField, actions); } private Optional\u0026lt;String\u0026gt; createShortCode(ShortenRequest req) { try { return Optional.ofNullable( req.alias() == null || req.alias().isBlank() ? store.createMapping(req.url()).shortCode() : store.createCustomMapping(req.alias(), req.url()).shortCode() ); } catch (IllegalArgumentException e) { return Optional.empty(); } } private record ShortenRequest(String url, String alias) { public ShortenRequest() { this(\u0026#34;\u0026#34;, \u0026#34;\u0026#34;); } public String url() { return url; } public String alias() { return alias; } } } This view encapsulates the entire user interaction for creating new short links. Incorrect entries result in immediate feedback, while valid entries are persisted and output directly as feedback.\n5.3 OverviewView: Grid-based management view # The OverviewView displays all existing mappings in the grid. Optionally, additional columns such as creation time or expiration date can be displayed. Each row allows deletion via a confirmation dialogue:\n@Route(value = \u0026#34;overview\u0026#34;, layout = MainLayout.class) public class OverviewView extends VerticalLayout { private final UrlMappingStore store = UiServiceLocator.getUrlMappingStore(); private final Grid\u0026lt;ShortUrlMapping\u0026gt; grid = new Grid\u0026lt;\u0026gt;(ShortUrlMapping.class, false); public OverviewView() { setSizeFull(); setPadding(true); setSpacing(true); grid.addColumn(ShortUrlMapping::shortCode).setHeader(\u0026#34;Short Code\u0026#34;); grid.addColumn(ShortUrlMapping::originalUrl).setHeader(\u0026#34;Original URL\u0026#34;).setFlexGrow(1); grid.addColumn(m -\u0026gt; m.createdAt().toString()).setHeader(\u0026#34;Erstellt am\u0026#34;); grid.addComponentColumn(this::buildActionButtons).setHeader(\u0026#34;Aktionen\u0026#34;); grid.setItems(store.findAll()); grid.setSizeFull(); add(new H2(\u0026#34;Alle Kurzlinks\u0026#34;), grid); } private Component buildActionButtons(ShortUrlMapping mapping) { Button delete = new Button(\u0026#34;Delete\u0026#34;, e -\u0026gt; openConfirmDialog(mapping)); delete.addThemeVariants(ButtonVariant.LUMO_ERROR); return delete; } private void openConfirmDialog(ShortUrlMapping mapping) { Dialog dialog = new Dialog(); dialog.add(new Text(\u0026#34;Really delete short link? (\u0026#34; + mapping.shortCode() + \u0026#34;)\u0026#34;)); Button confirm = new Button(\u0026#34;Ja\u0026#34;, e -\u0026gt; { store.delete(mapping.shortCode()); grid.setItems(store.findAll()); dialog.close(); Notification.show(\u0026#34;Short link deleted.\u0026#34;); }); confirm.addThemeVariants(ButtonVariant.LUMO_PRIMARY); Button cancel = new Button(\u0026#34;Cancel\u0026#34;, e -\u0026gt; dialog.close()); dialog.add(new HorizontalLayout(confirm, cancel)); dialog.open(); } } The grid is updated automatically after deletions. Additional filter or export functions can be easily added based on this structure.\n5.4 Consistent behaviour for dialogues and notifications (with code) # In an administration interface for a security-critical service such as a URL shortener, the way the system handles user interactions is crucial. The response to inputs—be they correct, incorrect, or critical—must be predictable, explainable and context-appropriate. Particularly important are:\nimmediate, non-blocking success messages for valid actions, precise, field-oriented fault indications in case of incorrect entries, intention-confirming dialogues for potentially destructive operations such as deleting mappings. These three feedback types can be implemented entirely using Vaadin Flow\u0026rsquo;s built-in tools – specifically: notification, dialogue, and component binding via Binder.\nSuccess stories via Notification # After successful actions (e.g., creating a short link or deleting an entry), a discreet, automatically disappearing notification is displayed. This is non-blocking, clearly informs users of the success, and allows them to continue working seamlessly.\nNotification.show(\u0026ldquo;Short link created successfully.\u0026rdquo;, 3000, Notification.Position.TOP_CENTER);\nFor targeted feedback with content (e.g. the generated short code), an inline component or a copyable text block can be displayed alternatively:\nvar shortCodeField = new TextField(\u0026#34;Your short link\u0026#34;); shortCodeField.setValue(\u0026#34;https://short.ly/\u0026#34; + code); shortCodeField.setReadOnly(true); shortCodeField.setWidthFull(); add(shortCodeField); Notification.show(\u0026#34;You can now copy the link.\u0026#34;); Error messages through field-based validation withBinder # Failed inputs—such as invalid URLs or duplicate aliases—do not need to be displayed in general dialogues, but rather directly in the affected field. To do this, Vaadin uses the Binder mechanism, which binds validation rules directly to UI components:\nBinder\u0026lt;ShortenRequest\u0026gt; binder = new Binder\u0026lt;\u0026gt;(ShortenRequest.class); binder.forField(urlField) .asRequired(\u0026#34;URL must not be empty\u0026#34;) .withValidator(this::isValidHttpUrl, \u0026#34;Only valid HTTP(S) URLs allowed\u0026#34;) .bind(ShortenRequest::url, ShortenRequest::url); The method isValidHttpUrl checks, for example, with a simple Regex or URI.create(\u0026hellip;) Logic for syntactical validity. Errors are automatically displayed as error messages directly below the field, visually highlighted and focused:\nprivate boolean isValidHttpUrl(String url) { return url != null \u0026amp;\u0026amp; (url.startsWith(\u0026#34;http://\u0026#34;) || url.startsWith(\u0026#34;https://\u0026#34;)); } If an error occurs at the application level (e.g. alias already assigned), this can also be communicated via a global notification:\nNotification.show(\u0026#34;Alias already taken\u0026#34;, 3000, Notification.Position.MIDDLE) .addThemeVariants(NotificationVariant.LUMO_ERROR); Destructive actions only after explicit confirmation # When deleting a short link, resetting a counter, or performing similar irreversible actions, the operation must never be performed without confirmation. Vaadin\u0026rsquo;s dialogue component, combined with descriptive text and two explicit buttons, is ideal for this.\nExample from the OverviewView:\nprivate void openConfirmDialog(ShortUrlMapping mapping) { Dialog dialog = new Dialog(); dialog.setHeaderTitle(\u0026#34;Confirm deletion\u0026#34;); Span message = new Span(\u0026#34;Do you really want to delete the short link? (\u0026#34; + mapping.shortCode() + \u0026#34;)\u0026#34;); dialog.add(message); Button confirm = new Button(\u0026#34;Delete\u0026#34;, click -\u0026gt; { store.delete(mapping.shortCode()); grid.setItems(store.findAll()); dialog.close(); Notification.show(\u0026#34;Short link deleted.\u0026#34;, 3000, Notification.Position.TOP_CENTER) .addThemeVariants(NotificationVariant.LUMO_SUCCESS); }); Button cancel = new Button(\u0026#34;Abbrechen\u0026#34;, event -\u0026gt; dialog.close()); cancel.addThemeVariants(ButtonVariant.LUMO_TERTIARY); dialog.getFooter().add(cancel, confirm); dialog.open(); } Features of this implementation:\nAn explicit button triggers the action. Confirmation is dialogue-based with a cancel option. The change will be reflected immediately upon completion. Feedback is provided immediately via notification. Interaction and reuse # A central UX pattern of this interface is to standardise the behaviour of all interactions. This avoids surprises for users. Consistent behaviour means:\naction Reaction URL successfully shortened Notification, display of the short link Incorrect input Field marking, inline error message Alias already taken central notification with incorrect colouring Delete the short link Dialogue with two buttons Confirmed deletion Notification: Grid is reloading These patterns can be encapsulated component-based – for example, using utility methods for dialogue generation or notification factories, which can later also be adapted for logging or internationalisation.\n7. Deploying the UI in the WAR context # The user interface is not delivered as a separate application, but integrated into a central WAR file that contains both the REST API and the Vaadin-based UI. This monolithic approach is deliberately chosen: It enables a consistent deployment model on servlet containers such as Jetty , Tomcat or Undertow without relying on complex build pipelines, containerization, or framework integration.\n7.1 Servlet configuration for Vaadin inweb.xml # Vaadin Flow is traditionally registered via the servlet com.vaadin.flow.server.VaadinServlet. Since there is no automatic configuration (as with Spring Boot), the assignment is made explicitly in the deployment descriptor:\n\u0026lt;web-app xmlns=\u0026#34;http://xmlns.jcp.org/xml/ns/javaee\u0026#34; version=\u0026#34;4.0\u0026#34;\u0026gt; \u0026lt;!-- I require UI --\u0026gt; \u0026lt;servlet\u0026gt; \u0026lt;servlet-name\u0026gt;VaadinServlet\u0026lt;/servlet-name\u0026gt; \u0026lt;servlet-class\u0026gt;com.vaadin.flow.server.VaadinServlet\u0026lt;/servlet-class\u0026gt; \u0026lt;heat-param\u0026gt; \u0026lt;param-name\u0026gt;ui\u0026lt;/param-name\u0026gt; \u0026lt;param-value\u0026gt;shortener.ui.MainUI\u0026lt;/param-value\u0026gt; \u0026lt;/init-param\u0026gt; \u0026lt;load-on-startup\u0026gt;1\u0026lt;/load-on-startup\u0026gt; \u0026lt;/servlet\u0026gt; \u0026lt;servlet-mapping\u0026gt; \u0026lt;servlet-name\u0026gt;VaadinServlet\u0026lt;/servlet-name\u0026gt; \u0026lt;url-pattern\u0026gt;/*\u0026lt;/url-pattern\u0026gt; \u0026lt;/servlet-mapping\u0026gt; \u0026lt;/web-app\u0026gt; The entry**/*** as a URL pattern causes all paths not occupied by the REST API (e.g./shorten, /abc123) to be handled by the UI. The REST endpoints can still be accessed via explicit HttpServlet registrations with priority.\nAlternatively, manual division via filters or explicit exclusion lists is possible, for example, /api/* should be strictly reserved for the REST branch.\n7.2 Theme-Integration ohne Frontend-Toolchain # A big advantage of Vaadin Flow in server mode is that there is no Node.js, npm or Webpack. All components and resources are handled at the JVM level and delivered as servlet resources at runtime. Themes can be defined directly in the classpath:\n@Theme(value = \u0026#34;shortener-theme\u0026#34;) public class MainLayout extends AppLayout { ... } In the directory frontend/themes/shortener-theme/, then lies:\nstyles.css– own adjustments theme.json– Metadata and component binding The Maven plugin vaadin-maven-plugin automatically generates the corresponding frontend bundle during package or install, fully integrated into the WAR build, without the need for external tools.\nA typical Maven excerpt inshortener-war/pom.xml:\n\u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;com.vaadin\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;vaadin-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${vaadin.version}\u0026lt;/version\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;prepare-frontend\u0026lt;/goal\u0026gt; \u0026lt;goal\u0026gt;build-frontend\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; The use of build-frontend. This is only required during initial deployment or when making theme changes. During operation, everything is loaded from the WAR.\n7.3 Operation in Tomcat or Jetty # The resulting WAR file can be used in any Servlet 4.0+ compatible container for deployment. Recommended are:\nJetty 12 for embedded deployments or local tests Apache Tomcat 10.1+ for production-related scenarios Eclipse Servlet Container (e.g. Open Liberty) for modular applications Deployment is incredibly simple:\ncp target/shortener.war $CATALINA_BASE/webapps/\nAfter starting the container, the application can be accessed via the usual path:\nhttp://localhost:8080/shortener/\nREST endpoints (e.g., POST /shorten) and UI routes (/overview, /create) coexist. This results in a robust, clearly controlled deployment model:\nNo framework magic No container lock-in No build complexity 8. Conclusion and outlook # With the integration of a Vaadin Flow-based interface, our URL shortener not only receives an interactive user interface, but also a complete, maintainable and production-ready management layer – realised exclusively with funds from the JDK. The graphical UI serves not as a decorative accessory, but as an integral part of the system architecture. It enables the secure creation, analysis, and maintenance of short links – typically via the same infrastructure used for REST endpoints or background processes.\nThe decision, exclusively Core Java and Vaadin Flow, has proven to be viable: The separation of module boundaries remains clear, deployment takes place as a classic WAR file, and all interactions – from validations and feedback logic to permission models – can be implemented in a comprehensible and transparent manner. Instead of framework magic, convention-over-configuration, or annotation-based autowiring, the focus here is once again on understandable system design down to the details.\nParticularly noteworthy is the efficiency with which Vaadin Flow maps UI logic directly in the Java context. Developers retain control over state, access protection, and lifecycles – without relying on build toolchains, client routing, or third-party libraries. The associated simplification of the maintenance and security model is a significant advantage in the context of safety-critical infrastructure.\nThe system can also be further developed in the future without any architectural disruption. Possible next steps include:\nModularisation of the UI into reading and writing segments Asynchronous Event-Analysis via server push or WebSockets Connection to external security systems such as OAuth2 or LDAP Deployment on embedded Jetty for minimalist distributions Migration to native builds via GraalVM to reduce runtime overhead This makes the URL shortener suitable not only as a learning object or internal tool, but also as a blueprint for lean, robust and self-determined Java web applications – without vendor lock-in, without complex tooling dependencies and with the focus on Readability, transparency and control.\nHappy Coding\nSven\n","date":"15 August 2025","externalUrl":null,"permalink":"/posts/part-iii-webui-with-vaadin-flow-for-the-url-shortener/","section":"Posts","summary":"1. Introduction and objectives # The first two parts of this series established the theoretical and practical foundations of a URL shortener in pure Java. We discussed the semantic classification of short URLs, the architecture of a robust mapping system, and the implementation of a REST-based service based on the JDK HTTP server. These efforts resulted in a functional, modularly extensible backend that creates, manages, and efficiently resolves short links. However, a crucial component was missing-a visual interface for direct user interaction with the system. This interface is essential for tasks such as manual link creation, viewing existing mappings, and analysing individual redirects.\n","title":"Part III - WebUI with Vaadin Flow for the URL Shortener","type":"posts"},{"content":" 1. Introduction # Why REST integration in Vaadin applications should not be an afterthought # In modern web applications, communication with external services is no longer a special function, but an integral part of a service-oriented architecture. Even if Vaadin Flow, as a UI framework, relies on server-side Java logic to achieve a high degree of coherence between view and data models, the need to communicate with systems outside the application quickly arises. These can be simple public APIs—for example, for displaying weather data or currency conversions—as well as internal company services, such as license verification, user management, or connecting to a central ERP system.\nThe challenge here lies not in technical feasibility. Still, in structural embedding, REST calls should not appear \u0026ldquo;incidentally\u0026rdquo; in the view code, but rather be abstracted and controlled in a cleanly encapsulated service layer. Especially in safety-critical or highly available systems, it\u0026rsquo;s not just access that matters, but its failure behaviour, fallback strategies, and reusability.\n1. Introduction Why REST integration in Vaadin applications should not be an afterthought 2. Architecture overview Separation of UI, service and integration – best practices without framework ballast 3. HTTP-Client in Java Introduction to java.net.http.HttpClient as a modern, native solution 4a. Der REST-Service als Adapter Building a lean Java class for REST communication with object mapping 4b. Der REST-Endpoint in Core Java How to create an HTTP endpoint without frameworks with minimal effort Minimal example: REST endpoint that returns JSON What is happening here, and why does it make sense Outlook: Modular expansion of REST servers 5. Type-safe data models with records How Java Records provide clarity and security 6. Error handling and robustness Handling status codes, IO problems and termination conditions HTTP status codes as a starting point Dealing with IO errors Logging with Thought Retry and Timeout Control instead of surprise 7. Integration in die Vaadin-UI Example use in the view layer – synchronous and understandable UI response to errors Visible feedback without blocking interaction Asynchronous extension (optional) 8. Production-ready safeguards Authentication, time limits, retry logic and logging – what you should pay attention to 8.1 Authentication: Headers instead of framework magic 8.2 Time limits: Protection against hanging services 8.3 Retry strategies: controlled and limited 8.4 Logging and Correlation 8.5 Resilience through convention 9. Asynchronous extension with CompletableFuture 1. When and how to integrate non-blocking HTTP calls into UI workflows 2. Starting point: Blocking REST calls 3. Solution: Non-blocking via CompletableFuture 4. Error handling in the futures chain 5. Progress indicator and UI states 6. Combination with multiple requests 10. Conclusion and outlook 1. REST adapters as a stable bridge in service-oriented architectures 2. Outlook: REST in modular and service-oriented Vaadin applications 3. What remains? Vaadin Flow offers no built-in mechanism for this, and that\u0026rsquo;s a good thing. By deliberately avoiding magical abstractions or framework coupling (such as Spring RestTemplate or Feign Clients), it allows maximum control over REST integration. With Java 24 and the now mature java.net.http.HttpClient, all required building blocks are available directly in the Core JDK – performant, maintainable and framework-independent.\nThis chapter, therefore, marks the beginning of a series of articles that demonstrate how to connect a Vaadin Flow application to an external REST endpoint, without external dependencies, but with a clear focus on readability, architecture, and security. The goal is not only to write functioning HTTP calls, but also to understand REST as part of a clean, modular software architecture.\n2. Architecture overview # Separation of UI, service and integration – best practices without framework ballast # A clear architecture is the foundation of any maintainable application, regardless of whether it is monolithic, modular, or service-oriented. This is especially true for Vaadin Flow, as UI components are defined server-side in Java, thus eliminating any rigid boundaries between the underlying layers. This proximity between the user interface and backend logic is one of Vaadin\u0026rsquo;s most significant advantages, but it also presents an architectural challenge when integrating external systems.\nThe integration of a REST endpoint should therefore always be done via an intermediary service layer. This refers to a dedicated \u0026ldquo;adapter\u0026rdquo; that, on the one hand, encapsulates the connection to the external REST service and, on the other hand, provides a type-safe, stable API for the UI layer. The advantage is obvious: Changes to the format of the remote system or the protocol behaviour do not affect the application structure or the user interface—they remain isolated in the adapter module.\nThis separation can be elegantly implemented in a Vaadin project without additional frameworks. A standard layered structure consists of the following components:\nUI layer: It contains the Vaadin views and components that work exclusively with Java data objects. No network communication or JSON processing takes place here. Service shift: It mediates between the UI and integration, potentially coordinates multiple adapters, and is responsible for domain-specific business logic. This layer is unaware of the technical details of the REST protocol. Adapter layer: It provides the actual REST access. This includes HTTP calls, header management, timeout handling, JSON deserialization, and error evaluation. This layer has no UI classes or interactive logic. This modular, three-part structure enables a testable and extensible framework. It enables the targeted simulation of REST access in unit tests or the use of alternative implementations (e.g., for offline modes or mocking in the development system) – all without requiring refactoring of the UI code.\nEspecially in Java 24, this principle can be excellently complemented with records, sealed types, and functional interfaces, making the resulting architecture not only stable but also concise and modern. The following figure (not included) illustrates how the data flows between the layers, from the UI request to the service, to the adapter, and back – always clearly separated but tightly interlinked by typed interfaces.\nWith this foundation in mind, we will discuss the technical implementation of REST communication in the next chapter, specifically using the HttpClient , which has been part of the JDK since Java 11 and has now established itself as a high-performance, well-tested standard.\n3. HTTP-Client in Java # Introduction to java.net.http.HttpClient as a modern, native solution # Since Java 11, the java.net.http.HttpClient, a powerful, standards-compliant, and thread-safe HTTP client, is available and fully included in the JDK. In Java, this client has long been established as a reliable means of REST communication, especially in projects that deliberately avoid frameworks to achieve maximum control, portability, and predictability.\nIn contrast to previous solutions, such as HttpURLConnection , which were laborious and error-prone, the modern HttpClient features a straightforward, fluent-based API that supports both synchronous and asynchronous communication using CompletableFuture. It is resource-efficient, supports automatic redirects, HTTP/2, and can be equipped with configurable timeouts, proxies, and authentication.\nIn practice, the use of the HttpClient begins with its configuration as a reusable instance:\nHttpClient client = HttpClient.newBuilder() .connectTimeout(Duration.ofSeconds(5)) .version(HttpClient.Version.HTTP_2) .build(); The HttpClient is immutable and thread-safe. Once created, it can be used for any number of requests. This is especially crucial in server applications, as repeated regeneration would not only be inefficient but also potentially problematic for resource utilisation.\nFor the specific request, a HttpRequest that encapsulates all parameters such as URI, HTTP method, headers and optional body data:\nHttpRequest request = HttpRequest.newBuilder() .uri(URI.create(\u0026#34;https://api.example.com/data\u0026#34;)) .header(\u0026#34;Accept\u0026#34;, \u0026#34;application/json\u0026#34;) .GET() .build(); The execution is synchronous with:\nHttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString());\nAlternatively, the method sendAsync() can be used to realise non-blocking interactions based on CompletableFuture – a model that can be particularly useful for UI-oriented or reactive applications when blocking HTTP processing should be avoided.\nWhat makes this API particularly special is the strict separation of request construction, client configuration, and result handling—no hidden magic, reflection, or XML-based configuration is used. Control lies entirely with the developer, and the API is understandable and type-safe.\nThe return is always made via a HttpResponse , where the type T is determined by the selected BodyHandler—typically String, byte[], InputStream, or Void. This separation allows for efficient processing of both simple text responses and binary data.\nIn the next chapter, I\u0026rsquo;ll show how this HTTP infrastructure can be converted into a production-ready adapter class, including header handling, error checking, and JSON deserialization into Java records. The goal is not just functional communication, but maintainable, robust, and testable code.\n4a. Der REST-Service als Adapter # Building a lean Java class for REST communication with object mapping # After the technical basics of the HttpClient are established in Java 24, the central question arises: How can REST calls be encapsulated so that they remain reusable, extensible, and UI-independent? The answer lies in the adapter principle, explicitly implemented as a standalone service class that encapsulates access to a specific REST endpoint and provides a type-safe API for the application core.\nSuch an adapter is responsible for:\nStructure and execution of the HTTP request, Interpretation of the HTTP response code, Transformation of the response data (e.g. JSON) into a Java model, optional: logging, header management, error handling and retry logic. However, these tasks shouldn\u0026rsquo;t be performed in the UI layer or presenter classes. Instead, a final-declared Java class is recommended, whose methods are specifically designed to load or send specific domain-specific data objects – for example: fetchUserData() , submitOrder() , validateLicenseKey().\nA minimal adapter for a GET call with a JSON response could look like this:\nimport java.io.IOException; import java.net.URI; import java.net.http.*; import java.time.Duration; import com.fasterxml.jackson.databind.ObjectMapper; import com.example.model.DataObject; public final class ExternalApiService { private final HttpClient client; private final ObjectMapper mapper; private final URI endpoint; public ExternalApiService(String baseUrl) { this.client = HttpClient.newBuilder() .connectTimeout(Duration.ofSeconds(5)) .build(); this.mapper = new ObjectMapper(); this.endpoint = URI.create(baseUrl + \u0026#34;/data\u0026#34;); } public DataObject fetchData() throws IOException, InterruptedException { HttpRequest request = HttpRequest.newBuilder() .uri(endpoint) .header(\u0026#34;Accept\u0026#34;, \u0026#34;application/json\u0026#34;) .GET() .build(); HttpResponse\u0026lt;String\u0026gt; response = client.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() != 200) { throw new IOException(\u0026#34;Unexpected status code: \u0026#34; + response.statusCode()); } return mapper.readValue(response.body(), DataObject.class); } } The aim of this class is not to be flexible for any endpoint or generic API, but rather to be concrete and targeted to serve exactly one external use case. This clarity is an advantage, not a disadvantage: It facilitates refactoring, testability, and secure extension if, for example, new headers, authentication mechanisms, or error codes are introduced.\nThe class encapsulates the entire technical aspect of HTTP communication. It is entirely independent of the UI and has no views, components, or user interactions. It can therefore be easily integrated into JUnit tests—either directly or via interface derivation for mocking.\nThe record class used DataObject serves as a transfer object that is automatically deserialised from JSON by the ObjectMapper:\npublic record DataObject(String id, String value) {}\nThis combination of a concise object structure and a stable communication layer forms the backbone of a clean REST integration—without frameworks, without magic, but with a clear separation of responsibilities. Errors can be specifically intercepted, additional parameters can be easily incorporated, and the architecture remains transparent.\n4b. Der REST-Endpoint in Core Java # How to create an HTTP endpoint without frameworks with minimal effort # A REST endpoint is essentially nothing more than a specialised HTTP handler. While Spring or JakartaEE use controllers, annotations, and automatic serialisation, a pure Java approach requires a bit more manual work, but is rewarded with complete control over behaviour, performance, security boundaries, and dependencies.\nSince Java 18, the com.sun.net.httpserver.HttpServer API, a lightweight HTTP server, has been available. It is ideal for simple REST endpoints—for example, as a test mock, an internal microservice, or in this case, as a local data source for Vaadin. In combination with a JSON mapping (e.g.ObjectMapper from Jackson), it creates a REST backend without any external platform.\nMinimal example: REST endpoint that returns JSON # The following class starts an HTTP server on port 8080, which, in a GET on /data, returns a JSON object:\nimport com.sun.net.httpserver.HttpServer; import com.sun.net.httpserver.HttpHandler; import com.sun.net.httpserver.HttpExchange; import com.fasterxml.jackson.databind.ObjectMapper; import java.io.IOException; import java.io.OutputStream; import java.net.InetSocketAddress; public class SimpleRestServer { public static void main(String[] args) throws IOException { HttpServer server = HttpServer.create(new InetSocketAddress(8080), 0); server.createContext(\u0026#34;/data\u0026#34;, new DataHandler()); server.setExecutor(null); // default executor server.start(); System.out.println(\u0026#34;Server is running on http://localhost:8080/data\u0026#34;); } static class DataHandler implements HttpHandler { private final ObjectMapper mapper = new ObjectMapper(); @Override public void handle(HttpExchange exchange) throws IOException { if (!\u0026#34;GET\u0026#34;.equals(exchange.getRequestMethod())) { exchange.sendResponseHeaders(405, -1); // Method Not Allowed return; } var data = new DataObject(\u0026#34;abc123\u0026#34;, \u0026#34;42.0\u0026#34;); String response = mapper.writeValueAsString(data); exchange.getResponseHeaders().add(\u0026#34;Content-Type\u0026#34;, \u0026#34;application/json\u0026#34;); exchange.sendResponseHeaders(200, response.getBytes().length); try (OutputStream os = exchange.getResponseBody()) { os.write(response.getBytes()); } } } public record DataObject(String id, String value) {} } What is happening here, and why does it make sense # The server listens on port 8080 and accepts requests/data in contrast to. Only GET requests are accepted; all other HTTP methods receive a status code of 405. The result object is an ObjectMapper-serialised instance – this is the same data type (DataObject) that is also used in the REST adapter on the client side. The answer will be delivered with the appropriate Content-Type, specifically application/json. This implementation is minimal but functional. It can be tested immediately, run locally, and queried from the Vaadin UI via the adapter described in Chapter 4a. It requires no XML, no DI container, no JAR hierarchies—just plain Java.\nOutlook: Modular expansion of REST servers # A REST server based on this principle can be easily expanded:\nMore createContext(\u0026hellip;)-Handler for additional endpoints Support for POST, PUT, DELETE with Payload-Parsing Path parameters through simple URI parsing Authentication logic via header evaluation Logging, metrics and request correlation as an extension Outsourcing to modules if multiple services are to be created This setup can be used particularly elegantly in development and testing, for example, to simulate external services or to reproduce specific error cases (404, 500).\n5. Type-safe data models with records # How Java Records provide clarity and security # The data model plays a central role in the communication between a Vaadin Flow application and an external REST service. It bridges the gap between the JSON representation of the remote resource and the object representation processed in Java. The clearer, more type-safe, and immutable this model is, the more robust the application architecture becomes, especially when the REST interface changes or when used in parallel UI contexts.\nSince Java 16, so-called Records – a language feature explicitly designed for this type of data structure: compact, immutable objects with semantically unambiguous data transport characteristics. A record is not a replacement for a complete domain entity with behaviour, but the ideal representation for structured response data from REST endpoints.\nAn example: A REST service delivers JSON responses like\n{ \u0026#34;id\u0026#34;: \u0026#34;abc123\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;42.0\u0026#34; } A suitable Java record to represent this structure looks like this:\npublic record DataObject(String id, String value) {}\nThis record is:\nUnchangeable : Its fields are final and publicly readable, but not modifiable. Comparable : equals() and hashCode() are automatically implemented correctly. Structured : The signature also serves as documentation of the expected JSON format. Compact : Compared to classic POJOs, there is no boilerplate code. Deserialization from JSON with an ObjectMapper(as shown in Chapter 4) works directly and without further annotations as long as the field names match. This promotes readability and reduces the risk of hidden deserialization errors.\nAdditionally, records are well-suited for use as DTOs (Data Transfer Objects). For example, when multiple external REST endpoints deliver different aspects of the same domain concept, each with its own response structure. Strict type safety of records then protects against unintentional mixing or incorrect field usage.\nRecords can also be nested to represent more complex JSON structures—for example, when an object contains a list of other objects or provides structured metadata. Here, too, developers benefit from the expressiveness, readability, and stability that records offer over traditional getter and setter classes.\nIn the context of a Vaadin application, this has an immediate positive effect: Records can be used directly in UI components, for example, for display in Grid , FormLayout or custom components. The data objects themselves contain no presentation logic, but remain purely structural—and this is precisely what is desirable in a separated architecture.\nIn the next chapter, we will demonstrate how this structured REST communication can be utilised in the UI layer and identify patterns that have proven successful in practice for effectively linking error handling, response logic, and user interaction.\n6. Error handling and robustness # Handling status codes, IO problems and termination conditions # In practice, a REST call is more than just an HTTP request. It is an insecure operation over a potentially unstable medium, with numerous possible sources of error, including network problems, timeout behaviour, invalid response formats, rejected requests, or temporary server errors. The quality of a REST integration is therefore evident, not in successful communication , but instead in its behaviour in the event of a mistake.\nEspecially in Vaadin Flow applications that remain active on the server side for extended periods, a single failed REST call can result in a view not being initialised correctly, a user action being suspended, or incomplete data being displayed. Therefore, the goal is to implement error handling that:\nsemantically distinguishes between transport-related and technical errors, provides defined return values ​​or exceptions to which the UI can react specifically, and established a comprehensible logging strategy. HTTP status codes as a starting point # Even the interpretation of the response code requires care. While 200 OK and 201 Created usually indicate success, other codes such as 204 No Content , 404 Not Found, or 409 Conflict can be entirely intentional – and should not be treated as errors across the board. The business context is crucial: A \u0026ldquo;not found\u0026rdquo; error can be deliberate and trigger a specific UI response, such as an empty display or an alternative form.\nThe adapter should therefore not perform generic error handling across all code, but rather explicitly evaluate what makes sense in the respective application scenario. Example:\nif (response.statusCode() == 404) { return Optional.empty(); // targeted reaction to missing resource } else if (response.statusCode() != 200) { throw new IOException(\u0026#34;Unexpected status: \u0026#34; + response.statusCode()); } Dealing with IO errors # Transport errors – such as timeouts, DNS problems, or connection failures – typically trigger IOException or InterruptedException. These should not be swallowed in the adapter or RuntimeException. Instead, the caller is signalled that the response is not usable. An error message can then be displayed in the UI without the application entering an undefined state.\nLogging with Thought # Errors that cannot be explained by user interaction should be logged on the server side, but not necessarily displayed to the user. A short Logger.warn(\u0026hellip;) An entry can help here, for example, to make external API errors traceable without flooding log files with irrelevant details. It is recommended to use different log levels depending on the error type: INFO for normal non-detectable behaviour, WARN for inconsistent responses, and ERROR only for real failures or bugs.\nRetry and Timeout # In production systems, it can be useful to automatically retry certain errors (e.g., 503 Service Unavailable or IOException) with backoff and limitation. However, this logic should not be implemented in the UI code, but rather in the adapter or a delegated \u0026ldquo;RetryHandler.\u0026rdquo; Timeouts should also be set explicitly:\nHttpRequest.newBuilder() .timeout(Duration.ofSeconds(3)) .build(); Without defined time limits, a REST call can inadvertently lead to a blocking UI thread, especially when called synchronously.\nControl instead of surprise # The central goal of any error handling is to ensure the Predictability of behaviour : A REST adapter should always provide defined semantics—either a correct result object, a declared exception, or an optional error that is explicitly checked. No hidden null values, no logic in catch blocks, no silent terminations.\nThe resulting UI remains able to react in a controlled manner, whether by displaying notifications, activating a retry button, or switching to offline data. The result is not only a more robust architecture but also a better user experience.\nIn the next chapter, I will show how these REST results are integrated into the UI layer of a Vaadin application, including concrete examples of loading operations, state changes, and error messages in the user context.\n7. Integration in die Vaadin-UI # Example use in the view layer – synchronous and understandable # The clean separation between REST communication and UI logic is a central principle of maintainable software architecture. However, there must come a point where external data actually ends up in the user interface—be it in a table, a form, or as part of an interactive visualisation. The integration of the REST adapter into the Vaadin UI is ideally done in this case. synchronous, type-confident and consciously visible in the code – instead of hidden “magic” or automatic binding.\nAt the centre is a view, e.g., a class that is a VerticalLayout or Composite and inherits. The REST adapter can be called within this view, typically when constructing the view or during a targeted user interaction. The following example demonstrates a simple use case: When loading the view, a dataset should be retrieved from an external REST service and displayed.\npublic class DataView extends VerticalLayout { private final ExternalApiService apiService; public DataView() { this.apiService = new ExternalApiService(\u0026#34;https://api.example.com\u0026#34;); try { DataObject data = apiService.fetchData(); showData(data); } catch (IOException | InterruptedException e) { showError(e.getMessage()); } } private void showData(DataObject data) { add(new Span(\u0026#34;ID: \u0026#34; + data.id())); add(new Span(\u0026#34;Wert: \u0026#34; + data.value())); } private void showError(String message) { Notification.show(\u0026#34;Error loading data: \u0026#34; + message, 5000, Notification.Position.MIDDLE); } } This type of integration is deliberately kept simple, but it demonstrates a central principle: The REST adapter is an explicit component of view initialisation. There\u0026rsquo;s no hidden binding, no automatic magic, just traceable, step-by-step data flow. This not only facilitates debugging but also makes state transitions visible and controllable.\nUI response to errors # Visible feedback without blocking interaction # The integration of an external REST endpoint into a Vaadin Flow application must not only be technically reliable, but it must also, and most importantly, behave predictably and stably within the user interface. In production scenarios, it is perfectly normal for REST services to be temporarily unavailable, provide slow responses, or fail with an error status, such as 500 Internal Server Error, 403 Forbidden, or 429 Too Many Requests. What matters is not the error itself, but how it affects the user experience.\nIn the example shown, a non-blocking notification is used in the event of an error. It appears in the centre of the screen and briefly informs the user that the data loading failed. This approach is recommended for several reasons:\nThe user remains able to act :\nA notification doesn\u0026rsquo;t interrupt the interaction flow. No dialogue opens, no loading process freezes, and no navigation is blocked. The application remains fully usable, which is especially beneficial in multi-part views or for later, optional loading operations.\n**The error message is visible in context: **Instead of hiding technical details in logs or simply not displaying \u0026ldquo;something,\u0026rdquo; the user is actively informed about the error—transparently, but without technical depth. The information isn\u0026rsquo;t intended for analysis, but rather to correct expectations:“Something should have appeared here, but is currently unavailable.” _\n_\n**The application does not exit into an undefined state: **There\u0026rsquo;s no exception chaining into the UI thread, no empty interface without explanation, and no sudden switching to another route. The user experience remains consistent, ​​even in the event of an error.\nFor more sophisticated scenarios, further reaction patterns are also available:\n**Retry button or “Try again” action: **A button can be displayed directly below the error message to re-execute the REST call. This leaves repeatability up to the user—useful, for example, in cases of temporary network errors or rate-limited APIs.\n**Fallback display or partial information: **If previous data (e.g., from local storage or a previous session) is available, it can be displayed as a placeholder, along with a note that the current data cannot be loaded at the moment.\n**Visual placeholders or skeleton layouts: **Instead of just displaying empty components, a visual framework can be displayed – for example, grey bars that indicate the expected content but symbolise it:“These data are not yet available.” _\n_\n**Consistent error design across the entire application: **It\u0026rsquo;s recommended to establish a central component or utility class for displaying errors that\u0026rsquo;s used system-wide. This ensures consistent display regardless of whether a REST error occurs in the dashboard, a dialogue, or a detailed view.\nIn summary, this approach pursues one central goal: Errors may occur, but they should not take over the control flow. The UI remains in the hands of the user, not the infrastructure. This creates a resilient interface that looks professional and inspires trust, even when systems in the background aren\u0026rsquo;t working perfectly.\nAsynchronous extension (optional) # In some cases, asynchronous integration can be useful – for example, when dealing with long loading times or interactions that must not block. This can also be achieved with Vaadin and the HttpClient. A typical strategy is to load the data in a background thread and then use UI.access(\u0026hellip;) back to the UI thread:\nUI ui = UI.getCurrent(); CompletableFuture.supplyAsync(() -\u0026gt; { try { return apiService.fetchData(); } catch (Exception e) { throw new CompletionException(e); } }).thenAccept(data -\u0026gt; ui.access(() -\u0026gt; showData(data))) .exceptionally(ex -\u0026gt; { ui.access(() -\u0026gt; showError(ex.getCause().getMessage())); return null; }); This variant is not necessary, but it is helpful in scenarios with many concurrent REST requests or slow-responding APIs.\nIntegrating a REST adapter into a Vaadin flow view is best done in a structured, explicit, and UI-specific manner. The view is not responsible for constructing the request or interpreting the HTTP code—it only consumes the Java objects provided by the adapter. Errors are handled, states are modelled consciously, and the user interface remains responsive and understandable.\nIn the next chapter, we\u0026rsquo;ll take a look at production-ready extensions: authentication, header handling, retry strategies, and logging—everything that makes REST access robust and secure.\n8. Production-ready safeguards # Authentication, time limits, retry logic and logging – what you should pay attention to # A REST adapter that works reliably in development and test environments is far from production-ready. As soon as a Vaadin Flow application is embedded in real infrastructures—with access to external APIs, network latencies, security policies, and operational requirements—additional protection mechanisms must be established. These not only address error cases, but also, and most importantly,Non-fault cases under challenging conditions, such as authentication, latency, rate limiting, logging requirements, or the protection of sensitive data.\nThe following discussion demonstrates how production-ready REST adapters can be developed using the resources of the Java Development Kit (JDK).\n8.1 Authentication: Headers instead of framework magic # External APIs typically require authentication, which can be in the form of a static token, a Basic Auth combination, or an OAuth 2.0 bearer token. In production-based scenarios, multiple variants are often used simultaneously – for example, an API that expects separate public and admin keys.\nInstead of shifting this logic to the HTTP client construction, it is recommended to provide your helper methods in the adapter, for example:\nprivate HttpRequest.Builder authenticatedRequest(URI uri) { return HttpRequest.newBuilder() .you(s) .header(\u0026#34;Authorization\u0026#34;, \u0026#34;Bearer \u0026#34; + tokenProvider.getCurrentToken()); } This not only isolates the technical mechanism (header), but also abstracts the source of supply (e.g., token rotation, expiration time). The token itself can be renewed regularly, read from a secrets store, or dynamically calculated, without affecting the call point in the UI code.\n8.2 Time limits: Protection against hanging services # Without explicitly set time limits, the HttpClient theoretically has unlimited waiting time for a response, which can be fatal in server-side applications. To control latency and protect UI threads, timeouts should be set both in the client and per request:\nHttpClient client = HttpClient.newBuilder() .connectTimeout(Duration.ofSeconds(5)) .build(); HttpRequest request = HttpRequest.newBuilder() .timeout(Duration.ofSeconds(3)) .type(...) .GET() .build(); This separation allows global parameters to be differentiated from the behaviour of individual calls – e.g., for particularly sensitive endpoints or third-party interfaces with historically fluctuating response times.\n8.3 Retry strategies: controlled and limited # Not all errors mean that a request fails permanently. Temporary DNS problems can often be intercepted by targeted retries**,** which can resolve503 Service Unavailable errors or connectionless gateways – provided the retries are limited, staggered in time, and context-dependent.\nAn example of a simple Retry loop without an external library:\npublic DataObject fetchDataWithRetry() throws IOException { int attempts = 3; for (int i = 1; i \u0026lt;= attempts; i++) { try { return fetchData(); // normal method with HttpClient } catch (IOException | InterruptedException e) { if (i == attempts) throw new IOException(\u0026#34;Maximum attempts reached\u0026#34;, e); try { Thread.sleep(i * 500L); // linear backoff } catch (InterruptedException ie) { Thread.currentThread().interrupt(); throw new IOException(\u0026#34;Retry aborted\u0026#34;, ie); } } } throw new IllegalStateException(\u0026#34;Unreachable code\u0026#34;); } Retry logic must never grow uncontrollably, so that it performs logging and that it is only active in the case of explicitly temporary errors, not in the case of 401 , 403 or 404.\n8.4 Logging and Correlation # In production-ready systems, REST calls are a relevant component of auditing, error analysis, and performance monitoring. Therefore, each call should be systematically recorded in the log, but in a differentiated manner:\nDEBUG : technical details such as URIs, headers, and response sizes INFO : regular REST call with semantic meaning (e.g. license check, status change) WARN : temporary errors or unexpected responses ERROR : permanent errors, incomplete response data, uncaught exceptions Additionally, a correlation token should be included with each request, for example, as a UUID in the X-Correlation-ID-Header , allowing REST calls to be traced across multiple systems. This can also be encapsulated centrally in the adapter:\nprivate HttpRequest.Builder withCorrelation(HttpRequest.Builder builder) {\n** return builder.header(\u0026ldquo;X-Correlation-ID\u0026rdquo;, UUID.randomUUID().toString());**\n}\n8.5 Resilience through convention # A stable REST adapter is not “intelligent” in the sense of being dynamic, but somewhat predictable, complete, and conservative. Every response is either a result or an exception—never a silence. The data formats are stable and defensively parsed. If expectations are not met, a defined termination occurs. This protects both the UI logic and the user experience, forming the basis for maintainable systems with precise error semantics.\n9. Asynchronous extension with CompletableFuture # When and how to integrate non-blocking HTTP calls into UI workflows # In Vaadin Flow applications, the code is traditionally structured synchronously – views respond to user actions, load data, and display results. However, as soon as REST calls come into play, which can take longer than a few milliseconds, the question inevitably arises: How can I offload HTTP communication without blocking the UI thread – and without having to resort to a reactive framework?\nThe answer to this is provided by the JDK integrated class CompletableFuture. It allows you to declaratively model asynchronous workflows, precisely control concurrency, and return the result to the UI in a controlled manner upon success or failure. In conjunction with Vaadin\u0026rsquo;s server-side UI model, only one measure is crucial: access to UI components must occur in the correct thread context.\nStarting point: Blocking REST calls # In a synchronous example, the call typically looks like this:\ntry { DataObject data = apiService.fetchData(); // synchronous call showData(data); // direct display in the UI } catch (IOException | InterruptedException e) { showError(e.getMessage()); } As long as response times are low and the view is already being constructed synchronously, this is entirely unproblematic. It becomes critical when REST calls follow user actions—for example, after clicking \u0026ldquo;Refresh\u0026rdquo; or during filter operations on large data sets. Here, a blocking call would lead to a noticeable delay in the UI.\nSolution: Non-blocking via CompletableFuture # To offload the REST call without blocking the UI, the access can be moved to a separate background thread, and the result can be safely returned to the UI later:\nUI ui = UI.getCurrent(); CompletableFuture.supplyAsync(() -\u0026gt; { try { return apiService.fetchData(); } catch (IOException | InterruptedException e) { throw new CompletionException(e); } }).thenAccept(data -\u0026gt; ui.access(() -\u0026gt; showData(data))) .exceptionally(ex -\u0026gt; { ui.access(() -\u0026gt; showError(ex.getCause().getMessage())); return null; }); What is happening here is architecturally remarkable:\nsupplyAsync(\u0026hellip;) starts the call in a separate thread of the Common ForkJoinPool. The fetchData() runs independently of the UI thread – does not block it. The result (or error) is then sent via UI.access(\u0026hellip;) and brought back into the server-side context where Vaadin UI components can be safely manipulated. This separation of calculation (REST call) and presentation (Vaadin components) corresponds to a classic principle of responsive design, only on the server-side level.\nError handling in the futures chain # Another advantage: Error handling is modelled clearly and separately, via**. exceptionally(\u0026hellip;)** The original exception object is available within this method. In many cases, a targeted display for the user is sufficient; ​​however, logging, metric collection, or retry logic can be added if necessary.\nProgress indicator and UI states # In asynchronous scenarios, it\u0026rsquo;s recommended to also provide visual feedback about the loading status—for example, through a spinner, a progress bar, or the targeted deactivation of interaction elements. Example:\nButton refresh = new Button(\u0026#34;Neu laden\u0026#34;); refresh.setEnabled(false); add(new ProgressBar()); CompletableFuture.supplyAsync(() -\u0026gt; apiService.fetchData()) .thenAccept(data -\u0026gt; ui.access(() -\u0026gt; { showData(data); refresh.setEnabled(true); })) .exceptionally(ex -\u0026gt; { ui.access(() -\u0026gt; { showError(\u0026#34;Loading failed\u0026#34;); refresh.setEnabled(true); }); return null; }); Combination with multiple requests # Parallel REST calls can also be made by CompletableFuture.allOf(\u0026hellip;) Orchestrate. Data from multiple services can be loaded independently and then analysed together. This technique is beneficial for dashboards, complex forms, or multi-API compositions.\nConclusion of this chapter:\nCompletableFuture provides an elegant way to integrate REST calls into Vaadin in a non-blocking manner, without any additional libraries. The control is type-safe and has minimal overhead. Combined with Vaadin\u0026rsquo;s UI.access(\u0026hellip;) mechanism, this creates reactive yet deterministic behaviour that is both technically and ergonomically compelling.\n10. Conclusion and outlook # REST adapters as a stable bridge in service-oriented architectures # The integration of external REST services into Vaadin Flow applications is far more than a technical detail. It is a conceptual component of a modular, maintainable, and externally communicative application. This blog post has shown how a REST connection without framework dependencies can be implemented in a structured and robust manner, purely using the onboard resources of the Java Core JDK from version 11, in idiomatic form for Java 24.\nThe focus is on an adapter that has three key features:\nTechnical clarity: The adapter takes full responsibility for handling HTTP requests, deserialising JSON, error handling, and optional authentication. The UI layer remains completely decoupled from it.\nType safety and predictability: The response data is cast into records – compact, immutable data structures that ensure readability, consistency and clean debugging.\nError robustness and extensibility: With time limits, logging, retry strategies and asynchronous access, the adapter can be made production-ready – without magic, but with conscious design.\nWhat initially starts as a simple way to “just add an HTTP call” to an application becomes architecturally viable communication module This approach is particularly advantageous for applications that grow over time: The REST adapter can evolve into a module that bundles various external systems, maintains version control, remains testable, and still interacts cleanly with the UI.\nOutlook: REST in modular and service-oriented Vaadin applications # With increasing complexity and system size, the need to distribute responsibilities grows: Data storage, business logic, UI, and integrations should be independently deployable, testable, and versionable. A typical architecture then moves toward:\nBackend-first architecture with Vaadin as UI facade, Microservice tailoring , where each external service is modelled as a port/adapter combination, Domain-centric UI , where views explicitly receive data via REST-controlled use cases, Security-by-Design , where REST communication is secured via tokens, headers and protocol standardisation. In all these scenarios, the REST adapter remains a central link – lean, deliberately modelled, but with precise semantics. Precisely because Vaadin Flow doesn\u0026rsquo;t attempt to automatically abstract or generically bind REST, the developer retains maximum control over requests, data structures, lifecycles, and user interaction.\nWhat remains? # Anyone building a Vaadin application that consumes REST should not treat REST as a workaround or side path, but as an equal interface to the world outside the UI. A stable, testable REST connection is a quality feature. And it can be achieved in Core Java with surprisingly few resources – if it is implemented consciously, structured, and in transparent layers.\nHappy Coding\n","date":"24 June 2025","externalUrl":null,"permalink":"/posts/connecting-rest-services-with-vaadin-flow-in-core-java/","section":"Posts","summary":"1. Introduction # Why REST integration in Vaadin applications should not be an afterthought # In modern web applications, communication with external services is no longer a special function, but an integral part of a service-oriented architecture. Even if Vaadin Flow, as a UI framework, relies on server-side Java logic to achieve a high degree of coherence between view and data models, the need to communicate with systems outside the application quickly arises. These can be simple public APIs—for example, for displaying weather data or currency conversions—as well as internal company services, such as license verification, user management, or connecting to a central ERP system.\n","title":"Connecting REST Services with Vaadin Flow in Core Java","type":"posts"},{"content":"","date":"24 June 2025","externalUrl":null,"permalink":"/tags/rest/","section":"Tags","summary":"","title":"REST","type":"tags"},{"content":" 1. Introduction to implementation # 1.1 Objectives and differentiation from the architectural part # The first part of this series focused on theory: We explained why a URL shortener is not just a convenience tool, but a security-relevant element of digital infrastructure. We discussed models for collision detection, entropy distribution, and forwarding logic, as well as analysed architectural variants – from stateless redirect services to domain-specific validation mechanisms.\n1. Introduction to implementation 1.1 Objectives and differentiation from the architectural part 1.2 Technological guardrails: WAR, Vaadin, Core JDK 1.3 Overview of the components 2. Project structure and module organisation 2.1 Structure of a modular WAR project 2.2 Separation of domain, API and UI code 2.3 Tooling und Build (Maven, JDK 24, WAR-Plugin) 3. URL encoding: Base62 and ID generation 3.1 Design of a stable short link scheme 3.2 Implementation of a Base62 encoder 3.3 Alternatives: Random, Hashing, Custom Aliases 3.4 Implementation: Base62Encoder.java 3.5 What is happening here – and why? 4. Mapping Store: Storage of the mapping 4.1 Interface-Design: UrlMappingStore 4.2 In-memory implementation with ConcurrentHashMap 4.3 Extensibility for later persistence 4.4 Implementation 4.5 Why so? 5. HTTP API with Java tools 5.1 HTTP-Server mit com.sun.net.httpserver.HttpServer 5.2 POST /shorten: Shorten URL 5.3 GET /{code}: Redirect to the original URL 5.4 Implementation Starting point: ShortenerServer.java POST Handler: ShortenHandler.java GET-Handler: RedirectHandler.java JsonUtils.java (Minimal Java JSON without external libraries) 5.6 Core Java Client Implementation 5.7 Summary This second part now turns to the concrete implementation. We develop a first working version of a URL shortener in Java 24 , consciously without the use of frameworks such as Spring Boot or Jakarta EE. The goal is to achieve a transparent, modularly structured solution that provides all core functions: URL shortening, secure storage of mappings, HTTP forwarding, and optional presentation via a Vaadin-based user interface.\nParticular attention is paid to the clean separation between encoding, storage, API, and UI. The entire application is delivered as a monolithic artefact – specifically, a classic WAR file , which is compatible with standard servlet containers such as Jetty or Tomcat. This decision enables rapid deployment and facilitates onboarding and testability.\n1.2 Technological guardrails: WAR, Vaadin, Core JDK # The implementation is based on a modern, yet deliberately lean technology stack. Only the built-in tools of the JDK and Vaadin Flow are used as the UI framework. The decision to use Vaadin is based on the requirement to implement interactive administration interfaces without additional JavaScript or separate front-end logic, entirely in Java.\nThe project is a multi-module structure. The separation between core logic, API layer, and UI remains visible and maintainable in the code. Maven is used as the build tool, supplemented by a WAR packaging plugin that creates a classic servlet deployment structure. The use of Java 24 enables the utilisation of modern language tools, including records, pattern matching, sequenced collections, and virtual threads.\nThe goal is a production-oriented, comprehensible implementation that can be used both as a learning resource and as a starting point for further product development.\n1.3 Overview of the components # The application consists of the following core components:\nA Base62-Encoder , which transforms consecutive IDs into URL-compatible short forms A Mapping-Store , which manages the mapping between the short link and the original URL A REST service , which allows URL shortening and resolution via redirect One optional UI based on Vaadin Flow for manual management of mappings\nand a configurable WAR deployment that integrates all components The architecture follows the principle: “As little as possible, as much as necessary.” Each part of the application is modular and allows for later splitting if necessary – for example, into separate services for reading, writing or analysis.\nIn the next chapter, we will focus on the concrete project structure and the module structure.\n2. Project structure and module organisation # 2.1 Structure of a modular WAR project # The first executable version of the URL shortener is realised as a monolithic Java application, which is in the form of a classic WAR.The project\u0026rsquo;s structure is based on a clear,layered architecture , which is prepared for later decomposition. The project is organised modularly, distinguishing between core logic, HTTP interface, and user interface. This separation not only allows for better maintainability but also forms the basis for the service decomposition planned in Part III or IV.\nThe project consists of three main modules:\nshortener-core: Contains all business logic, including URL encoding, data model and store interfaces. shortener-api: Implements the REST API based on the Java HTTP server (com.sun.net.httpserver.HttpServer). shortener-ui-required: Optional UI module with Vaadin Flow for managing and visualising mappings. These modules are distributed via a central WAR project (shortener war), which handles the delivery configuration and combines all dependencies. The WAR project is the only one that handles servlet-specific aspects (e.g., web.xml, I require a Servlet) – the remaining modules remain entirely independent of it.\n2.2 Separation of domain, API and UI code # 2.2 Separation of Domain, API, and UI Code\nThe modularisation of the project is based on the principle of technological isolation: The core business logic must know nothing about HTTP, servlet containers, or UI frameworks. This way, it remains fully testable, interchangeable, and reusable—for example, for future CLI or event-based variants of the shortener.\nThe core module defines all central interfaces (UrlMappingStore, ShortCodeEncoder) as well as the base classes (ShortUrlMapping, Base62Encoder). These components do not contain any I/O logic.\nThe api module is responsible for parsing HTTP requests, routing, and generating redirects and JSON responses. It accesses the core logic internally but remains detached from UI aspects.\nThe ui-vaadin module uses Vaadin Flow to implement a web-based interface, integrates the core logic directly, and is initialised in the WAR via a dedicated servlet definition.\nAdditional modules can be optionally added—for example, for persistence, monitoring, or analysis—without compromising the coherence of the structure.\n2.3 Tooling und Build (Maven, JDK 24, WAR-Plugin) # The build system is based on the current version of Maven. Each module is managed as a standalone Maven project with its pom.xml, with shortener-war configured as the parent WAR application. WAR packaging is handled using the standard servlet model, allowing the resulting file to be easily deployed in Tomcat, Jetty, or any Servlet 4.0+ compatible container.\nJava 24 is required at runtime, which is particularly relevant for modern language features such as record, pattern matching, and SequencedMap. Release 21 or higher is recommended as the target platform to ensure compatibility with modern runtimes.\nVaadin Flow integration is handled purely on the server side via the Vaadin Servlet and does not require a separate front-end build pipeline. Resources such as themes and icons are loaded entirely from the classpath.\n3. URL encoding: Base62 and ID generation # 3.1 Design of a stable short link scheme # The key requirement for a URL shortener is to generate unique, shortest possible character strings that serve as keys for accessing the original URL. To meet this requirement, the first implementation utilises a sequential ID scheme that assigns a consecutive numeric ID to each new URL. This ID is then converted into a URL-compatible format—specifically, Base62.\nBase62 includes the 26 uppercase letters, the 26 lowercase letters, and the 10 decimal digits. Unlike Base64, Base62 does not contain special characters such as +, /, or =, making it ideal for URLs: The generated strings are readable, system-friendly, and easily transferable in all contexts.\nThe resulting scheme is thus based on a two-step process:\nAssigning a unique numeric ID (e.g., 1, 2, 3, \u0026hellip;) Converting this ID to a Base62 string (e.g., 1 → b, 2 → c, \u0026hellip;) This method guarantees unique and unguessable codes, especially if the ID count does not start at 0 or if codes are additionally shuffled.\n3.2 Implementation of a Base62 encoder # The Base62-Encoder is used as a standalone utility class in the core module. It contains two static methods:\nencode(long value): converts a positive integer to a Base62 string decode(String input): converts a Base62 string back to an integer The alphabet is defined internally as a constant character string, and the conversion process is carried out purely mathematically, comparable to the representation of a number in another place value system.\nThis implementation creates a deterministic, stable, and thread-safe encoder that requires no external libraries. The resulting codes are significantly shorter than the underlying decimal number and contain no special characters—a key advantage for embedded or typed links.\nFor exceptional cases—such as custom aliases—the encoder remains optional, as such aliases can be stored directly as separate strings. However, by default, the Base62 encoder is the preferred method.\n3.3 Alternatives: Random, Hashing, Custom Aliases # In addition to the sequential approach, there are other methods for generating short links that can be considered in later stages of development:\nRandom-based tokens (z. B. UUID, SecureRandom) increase unpredictability, but require collision detection and additional memory overhead. The hashing process (e.g., SHA-1 of the destination URL) guarantees stability but is prone to collisions under high load or identical destination addresses. Custom aliases enable readable, short links (e.g., /helloMax), but require additional checking for collisions, syntactic validity, and protection of reserved terms. For the first version, we focus on the sequential model with Base62 transformation – a stable and straightforward approach.\n3.4 Implementation: Base62Encoder.java # The goal is to provide a simple utility class that converts integers to Base62 strings and vice versa. This class is thread-safe, stateless, and implemented without any external dependencies.\nFirst, the complete source code:\npublic final class Base62Encoder { private static final String ALPHABET = \u0026#34;0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\u0026#34;; private static final int BASE = ALPHABET.length(); private Base62Encoder() { } public static String encode(long number) { if (number \u0026lt; 0) { throw new IllegalArgumentException(\u0026#34;Only non-negative values supported\u0026#34;); } StringBuilder result = new StringBuilder(); do { int remainder = (int) (number % BASE); result.insert(0, ALPHABET.charAt(reminder)); number = number / BASE; } while (number \u0026gt; 0); return result.toString(); } public static long decode(String input) { if (input == null || input.isEmpty()) { throw new IllegalArgumentException(\u0026#34;Input must not be null or empty\u0026#34;); } long result = 0; for (char c : input.toCharArray()) { int index = ALPHABET.indexOf(c); if (index == -1) { throw new IllegalArgumentException(\u0026#34;Invalid character in Base62 string: \u0026#34; + c); } result = result * BASE + index; } return result; } } 3.5 What is happening here – and why? # This class encapsulates all Base62 encoding behaviour in two static methods. The character set consists of digits (0–9), lowercase letters (a–z), and uppercase letters (A–Z), resulting in exactly 62 different characters.\nThe method encode(long number) converts an integer into its inverse representation in the base 62 place value system. The remainder of the division by 62 is successively calculated, and the corresponding character is inserted. The result is a short, URL-friendly string.\nThe decode(String input) method reverses this process: it converts a Base62 string back into its numeric representation. Each character is replaced by its index in the alphabet and weighted accordingly.\nThis implementation is robust against invalid input, operates entirely in memory, and can be used directly in ID generation or URL mappings.\nImplementation: ShortCodeGenerator.java\nThe class abstracts ID generation from the concrete storage mechanism. It is suitable for both the in-memory version and future persistent variants. The generator is thread-safe and uses only JDK resources.\npublic final class ShortCodeGenerator { private final AtomicLong counter; public ShortCodeGenerator(long initialValue) { this.counter = new AtomicLong(initialValue); } public String nextCode() { long id = counter.getAndIncrement(); return Base62Encoder.encode(id); } public long currentId() { return counter.get(); } } Explanation\nThis class encapsulates a sequential counter that is incremented each time nextCode() generates a new, unique short code. The output is based on a monotonically increasing ID encoded into a Base62 string.\nThe getAndIncrement() method of AtomicLong is non-blocking and thus highly performant, even under high concurrency conditions. The generated code is unambiguous, compact, and deterministic—properties that are ideal for auditing, logging, and subsequent analysis.\nThe constructor allows you to configure the initial value. This is useful, for example, if you want to continue a persistent counter after a restart.\nExample usage (e.g. in the mapping store)\nShortCodeGenerator generator = new ShortCodeGenerator(1L); String shortCode = generator.nextCode(); // z. B. \u0026#34;b\u0026#34;, \u0026#34;c\u0026#34;, \u0026#34;d\u0026#34;, ... 4. Mapping Store: Storage of the mapping # 4.1 Interface-Design: UrlMappingStore # The mapping store forms the heart of the URL shortener. It manages the mapping between a short code (e.g.kY7zD) and the corresponding target URL (https://example.org/foo/bar). At the same time, it takes control over multiple uses, expiration times, and potential aliases.\nIn the first stage of development, a purely in-memory-based solution is used**.** This is fast, simple, and ideal for starting—even if it is lost upon reboot. Persistence is deliberately postponed to a later phase.\nThe store is abstracted via a simple interface. This interface allows for later substitution (e.g., with a file- or database-based version) without affecting the API or UI components.\n4.2 In-memory implementation with ConcurrentHashMap # The first concrete implementation utilises a ConcurrentHashMap to ensure reliable access even under heavy load. Each mapping entry is represented by a simple record object (ShortUrlMapping) that contains the target URL, creation time, and optional expiration information.\nThe combination of ConcurrentHashMap and ShortCodeGenerator allows deterministic and thread-safe ID assignment without the need for explicit synchronisation. This creates a high-performance solution that operates reliably even under high load.\n4.3 Extensibility for later persistence # All entries are accessible via a central interface. This interface is used not only for storage and retrieval but also forms the basis for later extensions, such as a persistence layer with flat files or EclipseStore, process control via TTL, or even event-driven backends.\nThe data structure can be extended with metrics, validity logic, or audit fields without requiring changes to the API – a classic approach to interface orientation in the sense of the open/closed principles.\n4.4 Implementation # public record ShortUrlMapping( String shortCode, String originalUrl, Instant createdAt, Optional\u0026lt;Instant\u0026gt; expiresAt ) {} This structure represents the basic assignment and allows the optional specification of an expiration time.\nStore-Interface: UrlMappingStore.java\npublic interface UrlMappingStore { ShortUrlMapping createMapping(String originalUrl); Optional\u0026lt;ShortUrlMapping\u0026gt; findByShortCode(String shortCode); boolean exists(String shortCode); List\u0026lt;ShortUrlMapping\u0026gt; findAll(); boolean delete(String shortCode); int mappingCount(); } The interface is deliberately kept slim and abstracts the two core operations: insert (with creation) and lookup.\nImplementation: InMemoryUrlMappingStore.java\npublic class InMemoryUrlMappingStore implements UrlMappingStore, HasLogger { private final ConcurrentHashMap\u0026lt;String, ShortUrlMapping\u0026gt; store = new ConcurrentHashMap\u0026lt;\u0026gt;(); private final ShortCodeGenerator generator; public InMemoryUrlMappingStore() { this.generator = new ShortCodeGenerator(1L); } @Override public ShortUrlMapping createMapping(String originalUrl) { logger().info(\u0026#34;originalUrl: {} -\u0026gt;\u0026#34;, originalUrl); String code = generator.nextCode(); ShortUrlMapping shortMapping = new ShortUrlMapping( code, originalUrl, Instant.now(), Optional.empty() ); store.put(code, shortMapping); return shortMapping; } @Override public Optional\u0026lt;ShortUrlMapping\u0026gt; findByShortCode(String shortCode) { return Optional.ofNullable(store.get(shortCode)); } @Override public boolean exists(String shortCode) { return store.containsKey(shortCode); } @Override public List\u0026lt;ShortUrlMapping\u0026gt; findAll() { return new ArrayList\u0026lt;\u0026gt;(store.values()); } @Override public boolean delete(String shortCode) { return store.remove(shortCode) != null; } @Override public int mappingCount() { return store.size(); } } 4.5 Why so? # The use of a ConcurrentHashMap ensures that concurrent write and read operations can be handled consistently and efficiently. The combination with AtomicLong in ShortCodeGenerator prevents collisions. The interface allows for a persistent implementation to be introduced later without changing the API or UI behaviour.\n5. HTTP API with Java tools # 5.1 HTTP-Server mit com.sun.net.httpserver.HttpServer # Instead of relying on heavyweight frameworks like Spring or Jakarta EE, we use the lightweight HTTP server implementation that the JDK already includes in the package com.sun.net.httpserver. This API, although rudimentary, is performant, stable, and perfectly sufficient for our use case.\nThe server is configured in just a few lines, requires no XML or annotation-based mappings, and can be controlled entirely programmatically. For each path, we define a separate HTTP handler that receives the request, processes it, and returns a structured HTTP response.\n5.2 POST /shorten: Shorten URL # The first endpoint allows a long URL to be passed over an HTTP POST to the shortener. In response, the server returns the generated short form, in the simplest case as a JSON object with the shortCode.\nExample request:\nPOST /shorten\nContent-Type: application/json\n{ \u0026#34;url\u0026#34;: \u0026#34;https://example.com/some/very/long/path\u0026#34; } Answer:\n200 OK\nContent-Type: application/json\n{ \u0026#34;shortCode\u0026#34;: \u0026#34;kY7zD\u0026#34; } Missing or invalid entries are marked with 400 Bad Request answered.\n5.3 GET /{code}: Redirect to the original URL # When calling a shortcode (e.g.GET /kY7zD), the server checks whether a mapping exists. If so, a HTTP 302 redirect to the original address. If the code is unknown or expired, a 404 Not Found error will be displayed.\nThis redirection is stateless and allows for later isolation into a read-only redirect service.\n5.4 Implementation # Starting point: ShortenerServer.java # public class ShortenerServer implements HasLogger { private HttpServer server; public static void main(String[] args) throws IOException { new ShortenerServer().init(); } public void heat() throws IOException { our store = new InMemoryUrlMappingStore(); this.server = HttpServer.create(new InetSocketAddress(8080), 0); server.createContext(\u0026#34;/shorten\u0026#34;, new ShortenHandler(store)); server.createContext(\u0026#34;/\u0026#34;, new RedirectHandler(store)); server.setExecutor(null); // default executor server.start(); System.out.println(\u0026#34;URL Shortener server running at http://localhost:8080\u0026#34;); } public void shutdown() { if (server != null) { server.stop(0); System.out.println(\u0026#34;URL Shortener server stopped\u0026#34;); } } } POST Handler: ShortenHandler.java # public class ShortenHandler implements HttpHandler, HasLogger { private final UrlMappingStorestore; public ShortenHandler(UrlMappingStore store) { this.store = store; } @Override public void handle(HttpExchange exchange) throws IOException { if (!\u0026#34;POST\u0026#34;.equalsIgnoreCase(exchange.getRequestMethod())) { exchange.sendResponseHeaders(405, -1); return; } InputStream body = exchange.getRequestBody(); Map\u0026lt;String, String\u0026gt; payload = parseJson(body); String originalUrl = payload.get(\u0026#34;url\u0026#34;); logger().info(\u0026#34;Received request to shorten url: {}\u0026#34;, originalUrl); if (originalUrl == null || originalUrl.isBlank()) { exchange.sendResponseHeaders(400, -1); return; } ShortUrlMapping mapping = store.createMapping(originalUrl); logger().info(\u0026#34;Created mapping for {} -\u0026gt; {}\u0026#34;, originalUrl, mapping.shortCode()); byte[] response = toJson(Map.of(\u0026#34;shortCode\u0026#34;, mapping.shortCode())).getBytes(StandardCharsets.UTF_8); exchange.getResponseHeaders().add(\u0026#34;Content-Type\u0026#34;, \u0026#34;application/json\u0026#34;); exchange.sendResponseHeaders(200, response.length); try (OutputStream os = exchange.getResponseBody()) { os.write(response); } } } GET-Handler: RedirectHandler.java # public class RedirectHandler implements HttpHandler , HasLogger { private final UrlMappingStorestore; public RedirectHandler(UrlMappingStore store) { this.store = store; } @Override public void handle(HttpExchange exchange) throws IOException { our requestURI = exchange.getRequestURI(); our fullPath = requestURI.getPath(); logger().info(\u0026#34;Full path: {}\u0026#34;, fullPath); String path = fullPath.substring(1); // strip leading \u0026#39;/\u0026#39; logger().info(\u0026#34;Path: {}\u0026#34;, path); if (path.isEmpty()) { exchange.sendResponseHeaders(400, -1); return; } Optional\u0026lt;String\u0026gt; target = store .findByShortCode(path) .map(ShortUrlMapping::originalUrl); if (target.isPresent()) { exchange.getResponseHeaders().add(\u0026#34;Location\u0026#34;, target.get()); exchange.sendResponseHeaders(302, -1); } else { exchange.sendResponseHeaders(404, -1); } } } JsonUtils.java (Minimal Java JSON without external libraries) # Since we do not want to use external dependencies such as Jackson or Gson in this first implementation, we need our own utility class to process simple JSON objects, specifically:\nString → Map\u0026lt;String, String\u0026gt;: for processing HTTP POST payloads (/shorten) Map\u0026lt;String, String\u0026gt; → JSON-String: to generate responses (e.g.{ \u0026ldquo;shortCode\u0026rdquo;: \u0026ldquo;abc123\u0026rdquo; }) This class is sufficient for simple key-value structures, as used in the shortener. It is not intended for nested objects or arrays , but as a pragmatic solution for the start.\nImplementation: JsonUtils.java\npublic final class JsonUtils { private JsonUtils() { } public static Map\u0026lt;String, String\u0026gt; parseJson(InputStream input) throws IOException { String json = readInputStream(input).trim(); return parseJson(json); } @NotNull public static Map\u0026lt;String, String\u0026gt; parseJson(String json) throws IOException { if (!json.startsWith(\u0026#34;{\u0026#34;) || !json.endsWith(\u0026#34;}\u0026#34;)) { throw new IOException(\u0026#34;Invalid JSON object\u0026#34;); } Map\u0026lt;String, String\u0026gt; result = new HashMap\u0026lt;\u0026gt;(); // Remove curly braces String body = json.substring(1, json.length() - 1).trim(); if (body.isEmpty()) { return Collections.emptyMap(); } // Separate key-value pairs with commas String[] entries = body.split(\u0026#34;,\u0026#34;); Arrays.stream(entries) .map(entry -\u0026gt; entry.split(\u0026#34;:\u0026#34;, 2)) .filter(parts -\u0026gt; parts.length == 2) .forEachOrdered(parts -\u0026gt; { String key = unquote(parts[0].trim()); String value = unquote(parts[1].trim()); result.put(key, value); }); return result; } public static String toJson(Map\u0026lt;String, String\u0026gt; map) { StringBuilder sb = new StringBuilder(); sb.append(\u0026#34;{\u0026#34;); boolean first = true; for (Map.Entry\u0026lt;String, String\u0026gt; entry : map.entrySet()) { if (!first) { sb.append(\u0026#34;,\u0026#34;); } sb.append(\u0026#34;\\\u0026#34;\u0026#34;).append(escape(entry.getKey())).append(\u0026#34;\\\u0026#34;:\u0026#34;); sb.append(\u0026#34;\\\u0026#34;\u0026#34;).append(escape(entry.getValue())).append(\u0026#34;\\\u0026#34;\u0026#34;); first = false; } sb.append(\u0026#34;}\u0026#34;); return sb.toString(); } private static String readInputStream(InputStream input) throws IOException { try (BufferedReader reader = new BufferedReader( new InputStreamReader(input, StandardCharsets.UTF_8))) { return reader.lines().collect(joining()); } } private static String unquote(String s) { if (s.startsWith(\u0026#34;\\\u0026#34;\u0026#34;) \u0026amp;\u0026amp; s.endsWith(\u0026#34;\\\u0026#34;\u0026#34;) \u0026amp;\u0026amp; s.length() \u0026gt;= 2) { return s.substring(1, s.length() - 1); } return s; } private static String escape(String s) { // simple escape logic for quotes return s.replace(\u0026#34;\\\u0026#34;\u0026#34;, \u0026#34;\\\\\\\u0026#34;\u0026#34;); } } Properties and limitations\nThis implementation is:\nfully JDK-based(no third-party libraries) for flat JSON objects suitable, i.e.{ \u0026ldquo;key\u0026rdquo;: \u0026ldquo;value\u0026rdquo; } robust against trivial parsing errors , but without JSON Schema validation Consciously minimalistic to stay within the scope of the prototype It is sufficient for:\nPOST /shorten(Client sends{ \u0026ldquo;url\u0026rdquo;: \u0026ldquo;\u0026hellip;\u0026rdquo; }) Response to this POST (server sends{ \u0026ldquo;shortCode\u0026rdquo;: \u0026ldquo;\u0026hellip;\u0026rdquo; }) Example use\nInputStream body = exchange.getRequestBody(); Map\u0026lt;String, String\u0026gt; input = JsonUtils.parseJson(body); String shortCode = \u0026#34;abc123\u0026#34;; String response = JsonUtils.toJson(Map.of(\u0026#34;shortCode\u0026#34;, shortCode)); For productive systems, the following is recommended in the future:\nGson(lightweight, idiomatic) Jackson(extensive, also for DTO binding) Json-B(Standard-API, Jakarta conform) However, for our first implementation in the Core JDK, the solution shown above deliberately remains the appropriate middle ground.\n5.6 Core Java Client Implementation # The class URLShortenerClient functions as a minimalist HTTP client for interacting with a URL shortener service. Its structure allows connection to a configurable or locally running server, with the default address being http://localhost:8080/. This enables easy integration into local development environments, test runs, or automated system tests without the need for additional configuration.\nAt the heart of the functionality is the method shortenURL(String originalUrl). It initiates an HTTP POST call against the server endpoint/shorten, transmits the URL to be shortened in a simple JSON document and immediately evaluates the server\u0026rsquo;s response. Successful completion is indicated exclusively by a status code.200 OK In this case, the method extracts the contained shortCode using the static auxiliary method extractShortCode() from the class JsonUtils. If the server returns a different HTTP code instead, the process will be aborted with a corresponding IOException, which enforces explicit error handling at the application level. This maintains a clear semantic separation between regular usage and exception situations.\nThe second central method,resolveShortcode(String shortCode), is used to explicitly resolve a short URL. It sends a GET request directly to the server\u0026rsquo;s root context, supplemented by the passed code. The behaviour of this method largely corresponds to that of a web browser, with the difference that automatic redirects have been deliberately deactivated. This way, the method can determine the actual target address, if present, ​​from the HTTP header field. Location and return it as a result. It clearly distinguishes between valid redirects (status 301 or 302), non-existent codes (status 404), and other unexpected responses. In the latter case, an IOException is thrown, analogous to the shortening process.\nTechnically speaking, the URLShortenerClient is exclusively composed of the Java SE API, namely HttpURLConnection, TYPE and stream-based input and output routines. All communication is UTF-8 encoded, ensuring high interoperability with modern JSON-based REST interfaces. The class also implements the interface HasLogger, which suggests a project-wide logging infrastructure and implicitly supports good traceability in server communication.\nThis client is particularly recommended for integration tests, command-line tools, or administrative scripts that require specific URLs to be shortened or verified. Due to its lean structure, the class is also suitable as a starting point for further abstractions, such as service-oriented encapsulation in larger architectures.\npublic class URLShortenerClient implements HasLogger { protected static final String DEFAULT_SERVER_PORT = \u0026#34;8080\u0026#34;; protected static final String DEFAULT_SERVER_URL = \u0026#34;http://localhost:\u0026#34; + DEFAULT_SERVER_PORT; protected static final String SHORTEN_URL_ENDPOINT = \u0026#34;/shorten\u0026#34;; protected static final String REDIRECT_URL_ENDPOINT = \u0026#34;/\u0026#34;; private final TYPE serverBase; public URLShortenerClient(String serverBaseUrl) { this.serverBase = TYPE.create(serverBaseUrl.endsWith(\u0026#34;/\u0026#34;) ? serverBaseUrl : serverBaseUrl + \u0026#34;/\u0026#34;); } public URLShortenerClient() { this.serverBase = TYPE.create(DEFAULT_SERVER_URL); } /** * String originalUrl = \u0026#34;https://svenruppert.com\u0026#34;; * * @param originalUrl * @return * @throws IOException */ public String shortenURL(String originalUrl) throws IOException { our serverURL = serverBase.toURL(); // --- Step 1: POST to the /shorten endpoint with a valid URL --- URL shortenUrl = URI.create(serverURL + SHORTEN_URL_ENDPOINT).toURL(); HttpURLConnection connection = (HttpURLConnection) shortenUrl.openConnection(); connection.setRequestMethod(\u0026#34;POST\u0026#34;); connection.setDoOutput(true); connection.setRequestProperty(\u0026#34;Content-Type\u0026#34;, \u0026#34;application/json\u0026#34;); String body = \u0026#34;{\\\u0026#34;url\\\u0026#34;:\\\u0026#34;\u0026#34; + originalUrl + \u0026#34;\\\u0026#34;}\u0026#34;; try (OutputStream os = connection.getOutputStream()) { os.write(body.getBytes()); } int status = connection.getResponseCode(); if (status == 200) { try (InputStream is = connection.getInputStream()) { String jsonResponse = new String(is.readAllBytes(), UTF_8); String extractedShortCode = JsonUtils.extractShortCode(jsonResponse); logger().info(\u0026#34;extractedShortCode .. {}\u0026#34;, extractedShortCode); return extractedShortCode; } } else { throw new IOException(\u0026#34;Server returned status \u0026#34; + status); } } public String resolveShortcode(String shortCode) throws IOException { logger().info(\u0026#34;Resolving shortCode: {}\u0026#34;, shortCode); URL url = URI.create(DEFAULT_SERVER_URL + REDIRECT_URL_ENDPOINT + shortCode).toURL(); logger().info(\u0026#34;url .. {}\u0026#34;, url); HttpURLConnection connection = (HttpURLConnection) url.openConnection(); connection.setInstanceFollowRedirects(false); int responseCode = connection.getResponseCode(); logger().info(\u0026#34;responseCode .. {}\u0026#34;, responseCode); if (responseCode == 302 || responseCode == 301) { our location = connection.getHeaderField(\u0026#34;Location\u0026#34;); logger().info(\u0026#34;location .. {}\u0026#34;, location); return location; } else if (responseCode == 404) { return null; } else { throw new IOException(\u0026#34;Unexpected response: \u0026#34; + responseCode); } } } 5.7 Summary # With just a few lines, you can create a functional HTTP API that serves as an ideal testbed and proof of concept. The structure is minimal but open to extensions such as error objects, rate limiting, or logging. What\u0026rsquo;s particularly remarkable is that the entire API works without servlet containers, external frameworks, or reflection—ideal for embedded applications and lightweight deployments.\nIn the next blog post, we will create a graphical interface to map user interactions.\nHappy Coding\nSven\n","date":"20 June 2025","externalUrl":null,"permalink":"/posts/part-ii-urlshortener-first-implementation/","section":"Posts","summary":"1. Introduction to implementation # 1.1 Objectives and differentiation from the architectural part # The first part of this series focused on theory: We explained why a URL shortener is not just a convenience tool, but a security-relevant element of digital infrastructure. We discussed models for collision detection, entropy distribution, and forwarding logic, as well as analysed architectural variants – from stateless redirect services to domain-specific validation mechanisms.\n","title":"Part II - UrlShortener - first Implementation","type":"posts"},{"content":"","date":"10 June 2025","externalUrl":null,"permalink":"/tags/architecture/","section":"Tags","summary":"","title":"Architecture","type":"tags"},{"content":"A URL shortener seems harmless – but if implemented incorrectly, it opens the door to phishing, enumeration, and data leakage. In this first part, I\u0026rsquo;ll explore the theoretical and security-relevant fundamentals of a URL shortener in Java – without any frameworks, but with a focus on entropy, collision tolerance, rate limiting, validity logic, and digital responsibility. The second part covers the complete implementation: modular, transparent, and as secure as possible.\n1.1 Motivation and use cases 1.2 Differentiation from related technologies 1.3 Objective of the paper 2.1 URI, URL and URN – conceptual basics 2.2 Principles of address shortening 2.3 Entropy, collisions and permutation spaces 4.2 URL Encoding: Hashing, Base62 and Alternatives 4.3 Mapping Store: Interface, Implementation, Synchronisation 4.4 REST API with pure Java (HTTP server, handler, routing) 4.5 Error handling, logging and monitoring 5.1 Abuse opportunities and protection mechanisms 5.2 Rate limiting and IP-based throttling 5.3 Validity period and deletion concepts 5.4 Protection against enumeration and information leakage 6.1 Access times and hash lookups 6.2 Memory usage and garbage collection 6.3 Benchmarking: Local tests and load simulation 7.1 Custom aliases 7.2 Access counting and analytics 7.3 QR-Code-Integration 7.4 Integration into messaging or tracking systems 8.1 Data protection for link tracking 8.2 Responsibility for forwarding 8.3 Transparency and disclosure of the destination address 9.1 Lessons Learned 9.2 Possible further developments (e.g. blockchain, DNSSEC) 9.3 Importance of URL shorteners in the context of digital sovereignty 1.1 Motivation and use cases # In an increasingly fragmented and mobile information world, URLs are not just technical addressing mechanisms; they are central building blocks of digital communication. Long and hard-to-remember URLs are a hindrance in social media, emails, or QR codes, as they are not only aesthetically unappealing but also prone to errors when manually entered. URL shorteners address this problem by generating compact representations that point to the original target address. In addition to improved readability, aspects such as statistical analysis, access control, and campaign tracking also play a key role.\nInitially popularised by services like TinyURL or bit.ly, URL shorteners have now become integrated into many technical infrastructures – from marketing platforms and messaging systems to IoT applications, where storage and bandwidth restrictions play a significant role. A shortened representation of URLs is also a clear advantage in the context of QR codes or limited character sets (e.g., in SMS or NFC data sets).\n1.2 Differentiation from related technologies # A URL shortener is not a classic forwarding platform and is conceptually different from proxy systems, link resolvers, or load balancers. While the latter often operate at the transport or application layer (Layer 4 or Layer 7 in the OSI model) and optimise transparency, availability, or performance, a shortener primarily pursues the goal of simplifying the display and management of URLs. Nevertheless, there are overlaps, particularly in the analysis of access patterns and the configuration of redirect policies.\nIn this work, a minimalist URL shortener is designed and implemented. It deliberately avoids external frameworks to implement the central concepts in a comprehensible and transparent manner in Core Java. The choice of Java 24 enables the integration of modern language features, such as records, sealed types, and virtual threads, into a secure and robust architecture.\n1.3 Objective of the paper # This paper serves a dual purpose: on the one hand, it aims to provide a deep technical understanding of the functionality and challenges associated with a URL shortener. On the other hand, it serves as a practical guide for implementing such a service using pure Java—that is, without Spring, Jakarta EE, or external libraries.\nTo this end, a comprehensive architecture will be developed, implemented, and continually enhanced with key aspects such as security, performance, and extensibility. The focus is deliberately on a system-level analysis of the processes to provide developers with a deeper understanding of the interaction between the network layer, coding strategies, and persistent storage. The goal is to develop a viable model that can be utilised in both educational contexts and as a basis for productive services.\n2. Technical background # 2.1 URI, URL and URN – conceptual basics # In everyday language, terms such as \u0026ldquo;URL\u0026rdquo; and \u0026ldquo;link\u0026rdquo; are often used synonymously, although in a technical sense, they describe different concepts.URI (Uniform Resource Identifier) refers to any character string that can uniquely name or locate a resource. A URL**(Uniform Resource Locator)** is a special form of a URI that not only identifies but also describes the access path, for example, through a protocol such as https, ftp, or mailto. A URN (Uniform Resource Name), on the other hand, names a resource persistently without referring to its physical address, such as urn:isbn:978-3-16-148410-0.\nIn the context of URL shorteners, URLs are exclusively concerned with accessible paths, typically via HTTP or HTTPS. The challenge is to transform these access paths in a way that preserves their semantics while reducing their representation.\n2.2 Principles of address shortening # The core idea of ​​a URL shortener is to replace a long URL string with a shorter key that points to the original address via a mapping. This mapping is done either directly in a lookup store (e.g., hash map, database table) or indirectly via a computational method (e.g., a hash function with collision management).\nThe goal is to use the redundancy of long URLs to map their entropy to a significantly shorter string. This poses a trade-off between collision-freeness, brevity, and readability. Conventional methods are based on encoding unique keys in a Base62 alphabet ([0-9a-zA-Z]), which offers 62 states per character. Just six characters can represent over 56 billion unique URLs—sufficient for many productive applications.\nThe shortcode acts as the primary key for address resolution. It is crucial that it is stable, efficiently generated, and as challenging to guess as possible to prevent misuse (e.g., brute-force enumeration).\n2.3 Entropy, collisions and permutation spaces # A key aspect of URL shortening is the question of how many different short addresses a system can actually generate. This consideration directly depends on the length of the generated shortcuts and their character set. Many URL shorteners use a so-called Base62 alphabet. This includes the ten digits from zero to nine, the 26 lowercase letters, and the 26 uppercase letters, for a total of 62 different characters.\nFor example, if you generate abbreviations with a fixed length of six characters, you get a combinatorial space in which over 56 billion different character strings are possible. Even with this relatively short number of characters, billions of unique URLs can be represented, which is more than sufficient for many real-world applications. For longer abbreviations, the address space grows exponentially.\nBut the sheer number of possible combinations is only one aspect. How these shortcuts are generated is equally important. If the generation is random, it is essential to ensure that no duplicate codes are created – so-called collisions. These can be managed either by checking for their existence beforehand or by deterministic methods such as hash functions. However, hash methods are not without risks, especially under heavy load: The more entries there are, the higher the probability that two different URLs will receive the same short code, especially if the hash function has not been optimised for this use case.\nAnother criterion is the distribution of the generated shortcuts. A uniform distribution in the address space is desirable because, on the one hand, it reduces the risk of collisions, and on the other hand, it increases the efficiency of storage and retrieval mechanisms – for example, in sharding for distributed systems or caching in high-traffic environments. Cryptographically secure random numbers or specially designed generators play a crucial role here.\nOverall, it can be said that the choice of alphabet, the length of the abbreviations and the way they are generated are not just technical parameters, but fundamental design decisions that significantly influence the security, efficiency and scalability of a URL shortener.\n3. Architecture of a URL shortener # The architecture of a URL shortener is surprisingly compact at its core, but by no means trivial. Although its basic function is simply to link a long URL with a short alias, numerous technical and conceptual decisions arise in the details. These include data storage, the structure of API access, concurrency behaviour, and security against misuse. This chapter explains the central components and their interaction, deliberately avoiding external frameworks. Instead, the focus is on a modular, transparent structure in pure Java.\nAt the heart of the system is a mapping table – typically in the form of a map or a persistent key-value database – that uniquely assigns each generated short code to its corresponding original URL. This structure forms the backbone of the shortener. Crucially, this mapping must be both efficiently readable and consistently modifiable, especially under load or when accessed concurrently by multiple clients.\nA typical URL shortener consists of three logically separate units: an input endpoint for registering a new URL, a redirection endpoint for evaluating a short link, and a management unit that provides metadata such as expiration times or access counters. In a purely Java-based solution without frameworks, network access is provided via the HTTP server introduced in Java 18. com.sun.net.httpserver package. This allows you to define REST-like endpoints with minimal overhead and to communicate with HttpExchange objects.\nThere are various options for storing mappings. In-memory structures, such as ConcurrentHashMap , offer maximum speed but are volatile and unsuitable for productive applications without a backup mechanism. Alternatively, file-based formats, relational databases, or object-oriented stores such as EclipseStore can be used. This paper will initially work with volatile storage to illustrate the basic logic. Persistence will be added modularly later.\nAnother key aspect concerns concurrency behaviour. Since URL shorteners are typically burdened by a large number of read accesses, for example, when calling short links, the architecture must be designed to allow concurrent access to the lookup table without locking conflicts. The same applies to the generation of new shortcuts, which must be atomic and collision-free. Java 24 introduces modern language tools, including virtual threads and structured concurrency, which can be utilised to manage server load in a more deterministic and scalable manner.\nLast but not least, horizontal extensibility plays a role. A cleanly decoupled design allows the shortener to be easily transferred to distributed systems later. For example, the actual URL resolver can be operated as a stateless service, while data storage is outsourced to a shared backend. Caching strategies and load balancing can also be integrated much more easily in such a setup.\nIn summary, a URL shortener is much more than a simple string replacement. Its architecture must be both efficient, robust, and extensible—properties that can be easily achieved through a modular structure in pure Java.\n4. Implementation with Java 24 # 4.1 Project structure and module overview # The implementation of the URL shortener follows a modular structure that supports both clarity in the source code and testability, as well as extensibility. The project is structured as a Java module and leverages the capabilities of the Java Platform Module System (JPMS). The goal is to separate the core functionality—that is, the management of URL mappings—from the network layer and persistence. This keeps the business logic independent of specific storage or transport mechanisms.\nAt the centre is a module called shortener.core , which contains all domain-specific classes: for example, the ShortUrlMapping, the UrlEncoder, as well as the central UrlMappingStore interface with a simple implementation in memory. A module shortener.http , which is based on Java\u0026rsquo;s internal HTTP server. It implements the REST endpoints and utilises the core module\u0026rsquo;s components for actual processing. Additional optional modules, such as those for persistence or analysis, can be added later.\nTo organise the code, a directory structure that clearly reflects the module and layer boundaries is recommended. Within the modules, a distinction should be made between api , impl , util and, if necessary, service.\n4.2 URL Encoding: Hashing, Base62 and Alternatives # A central element of the shortener is the mechanism for generating short, unique codes. This implementation uses a hybrid method that generates a consecutive, atomic sequence number and converts it into a human-readable format using a Base62 encoder.\nThis choice has two advantages: First, it is deterministic and avoids collisions without the need for complex hash functions. Second, generated codes can be efficiently serialised and are easy to read, which is particularly relevant in marketing or print contexts. Alternatively, cryptographic hashes such as SHA-256 can be used when unpredictability and integrity protection are essential, for example, for signed links or zero-knowledge schemes.\nThe Base62 encoder is implemented as a pure utility class that encodes integer values ​​into a character string, where the alphabet consists of numbers and letters. Inverse decoding is also provided in case bidirectional analysis is required in the future.\n4.3 Mapping Store: Interface, Implementation, Synchronisation # For managing URL mappings, a clearly defined interface called UrlMappingStore provides methods for inserting new mappings, resolving short links, and optionally managing metadata. The default implementation, InMemoryUrlMappingStore, is based on a ConcurrentHashMap and utilises AtomicLong for sequence number generation.\nThis simple architecture is completely thread-safe and allows parallel access without external synchronisation mechanisms. The implementation can be replaced at any time with a persistent variant, for example, based on flat file storage or through integration with an object-oriented storage system such as EclipseStore.\nThis separation keeps the application core stable while treating storage as a replaceable detail—a classic example of the dependency inversion principle in the spirit of Clean Architecture.\n4.4 REST API with pure Java (HTTP server, handler, routing) # The REST interface is implemented exclusively with the built-in tools of the JDK. Java provides the package com.sun.net.httpserver , which offers a minimalistic yet powerful HTTP server ideal for lean services. For the implementation of the API, a separate HttpHandler is defined that responds to specific routes, such as /shorten for POST requests and /{code} for forwarding.\nThe implementation is based on a clear separation between parsing, processing, and response generation. Incoming JSON messages are parsed manually or with the help of simple helper classes, without the need for external libraries. HTTP responses also follow a minimalist format, characterised by structured status codes, simple header management, and UTF-8-encoded bodies.\nRouting is handled by a dispatcher class, which selects the appropriate handler based on the request path and HTTP method. Later extensions, such as CORS, OPTIONS handling, or versioning, are easily possible.\n4.5 Error handling, logging and monitoring # In a productive environment, robust error handling is essential. The implementation distinguishes between systematic errors (such as invalid inputs or missing short codes) and unexpected runtime errors (such as IO problems or race conditions). The former are reported with clear HTTP status codes, such as 400 (Bad Request) or 404 (Not Found). The latter leads to a generic 500 Internal Server Erro r, with the causes being logged internally.\nFor logging, the JDK’s own java.util.logging This allows for platform-independent logging and can be replaced with SLF4J-compatible systems if needed. Monitoring metrics such as access counts, response times, or error statistics can be made accessible via a separate endpoint or JMX.\n5. Security aspects # 5.1 Abuse opportunities and protection mechanisms # A URL shortener can easily be used to obscure content. Attackers deliberately exploit the shortening to redirect recipients to phishing sites, malware hosts, or dubious content without the target address being immediately visible. This can pose significant risks, especially for automated distributions via social networks, chatbots, or email campaigns.\nAn adequate protection mechanism consists of automatically validating all target addresses upon insertion, for example, through syntactical URL checks, DNS resolution, and optionally through a background query (head request or proxy scan) that ensures that the target page is accessible and non-suspicious. Such checks should be modular so that they can be activated or deactivated depending on the environment (e.g., offline operation). Additionally, logging should be performed every time a short link is accessed, making it easier to identify patterns of abuse.\n5.2 Rate limiting and IP-based throttling # Another risk lies in excessive use of the service, be it through botnets, targeted enumeration, or simple DoS behaviour. A robust URL shortener should therefore have rate limiting that restricts requests within a given time slot. This can be global, IP-based, or per-user, depending on the context.\nIn a Java implementation without frameworks, this can be achieved, for example, via a ConcurrentHashMap that maintains a timestamp or counter buffer for each IP address. If a threshold is exceeded, the connection is terminated with a status code of 429 Too Many Requests rejected. This simple throttling can be supplemented with leaky bucket or token bucket algorithms if necessary to achieve a fairer distribution over time. For productive use, logging of critical threshold violations is also recommended.\n5.3 Validity period and deletion concepts # Not every short link should remain valid forever. A configurable validity period is essential, especially for security-critical applications, such as temporary document sharing or one-time authentication. A URL shortener should therefore offer the option of defining expiration times for each mapping.\nOn a technical level, it is sufficient to assign an expiration date to each mapping, which is checked during the lookup. When accessing expired short links, either an error status, such as 410 Gone, is displayed, or the user is redirected to a defined information page. Additionally, there should be periodic cleanup mechanisms that remove expired or unused entries from memory, such as through a time-controlled cleanup process or lazy deletion upon access.\n5.4 Protection against enumeration and information leakage # An often overlooked attack vector is the systematic scanning of the abbreviation space – for example, by automated retrieval of /aaaaaa until /zzzzzz. If a URL shortener delivers valid links without any protection mechanisms, potentially confidential information about the existence and use of links can be leaked.\nAn adequate protection consists in making the shortcuts themselves non-deterministic – for example, by using cryptographically generated, unpredictable tokens instead of continuous sequences. Additionally, access restrictions can be introduced, allowing only authenticated clients to access certain short links or excluding specific IP ranges. The targeted obfuscation of error responses – for example, by consistently issuing 404 Not Found even with blocked or expired abbreviations – makes analysis more difficult for attackers.\nA further risk arises when metadata such as creation time, number of accesses, or request origin is exposed unprotected via the API. Such information should only be accessible to authorised users or administrative interfaces and should never be part of the public API output.\n6. Performance and optimisation # 6.1 Access times and hash lookups # The most common operation in a URL shortener is resolving a shortcode into its corresponding original URL. Since this is a classic lookup operation, the choice of the underlying data structure is crucial. In the standard implementation, a ConcurrentHashMap, which is optimised in Java 24, has fine-grained locking. This offers nearly constant access times – even under high concurrency – and is therefore ideal for read-intensive workloads, such as those typical of a shortener.\nThe latency of such an operation is in the range of a few microseconds, provided the lookup table is stored in main memory and no additional network or IO layers are involved. However, if data storage is outsourced to persistent systems, such as a relational database or a disk-based key-value store, the access time increases accordingly. Therefore, it is recommended to cache frequently accessed entries – either directly in memory or via a dedicated cache layer.\nPerformance also plays a role in the creation of new abbreviations. This is where sequence number generation using AtomicLong is used, providing a thread-safe, low-contention solution for linear ID assignment. Combined with Base62 encoding, this creates a fast, predictable, and collision-free process.\n6.2 Memory usage and garbage collection # Since a URL shortener must manage a growing number of entries over a longer period, it is worthwhile to examine its storage behaviour. ConcurrentHashMap. While this results in fast access times, it also means that all active mappings remain permanently in memory—unless cleanup is implemented. A simple mapping structure consisting of a shortcode, original URL, and an optional timestamp requires several hundred bytes per entry, depending on the JVM configuration and string length.\nWith several million entries, heap usage can reach several gigabytes. To improve efficiency, care should be taken to use objects sparingly. For example, common URL prefixes (e.g. https://) are replaced with symbolic constants. Records instead of classic POJOs also help reduce object size and minimise GC load.\nIn the long term, it is recommended to introduce an active or passive cleanup mechanism, such as TTL-based eviction or access counters, to specifically remove rarely used entries. WeakReference or soft caching should be considered with caution, since the semantics of such structures do not always lead to expected behaviour in the server context.\n6.3 Benchmarking: Local tests and load simulation # Systematic benchmarking is essential for objectively evaluating the performance of a URL shortener. At a local level, this can be achieved with simple Java benchmarks that measure sequence number generation, lookup time, and code distribution quality. Tools such as JMH (Java Microbenchmark Harness) can also be used. Although external tools are not used in this paper, a manual microbenchmarking approach using System.nanoTime and a targeted warm-up can provide valuable insights.\nFor more realistic tests, a load simulation with HTTP clients is suitable, for example, using simple JDK-based multi-thread scripts or tools such as curl. In particular, behaviour under high concurrent access load should be observed, both in terms of response times and resource consumption. Behaviour in the event of failed requests, rapid-fire access, or expired links should also be explicitly tested.\nThe goal of such benchmarks is not only to validate the maximum transaction rate, but also to verify stability under continuous load. A robust implementation should not only be high-performance but also deterministic in its response behaviour and resistant to out-of-memory errors. Optional profiling—for example, using JDK Flight Recorder—can reveal further optimisation potential.\n7. Expansion options and variants # 7.1 Custom aliases # A frequently expressed wish in practice is the ability to not only use automatically generated short links, but also to assign custom aliases – for example, for marketing campaigns, internal documents, or individual redirects. A custom alias, such as /travel2025 is much easier to remember than a random Base62 token and can be integrated explicitly into communication and branding.\nTechnically speaking, this expands the mapping store\u0026rsquo;s responsibility. Instead of only accepting numerically generated keys, the API must verify that a user-defined alias is syntactically valid, not already in use, and not reserved. A simple regex check, supplemented by a negative list for reserved terms (e.g. /admin , /api), is sufficient to get started. This alias must then be treated equally to the automatically generated codes when stored.\nThis creates new failure modes, for example, when a user requests an alias that already exists. Such cases should be handled consistently with a 409 Conflict. The API can optionally suggest alternative names—a small convenience feature with a significant impact on the user experience (UX).\n7.2 Access counting and analytics # A functional URL shortener is more than just a redirection tool—it\u0026rsquo;s also an analytics tool. Tracking how often, when, and from where a short link was accessed is particularly relevant in the context of campaigns, product pages, or documented distribution.\nTo implement this functionality, each successful resolution of a short link must be saved as an event, ​​either by simply incrementing a counter or by fully logging with a timestamp, IP address, and user agent. For the in-memory variant, an additional AtomicLong or a metric structure aggregated via a map. Alternatively, detailed access data can be persisted in a dedicated log file or an external analytics module.\nThe evaluation can be performed either synchronously via API endpoints (e.g.,/stats/{alias}) or asynchronously via export formats such as JSON, CSV, or Prometheus metrics. Integration with existing logging systems (e.g. via java.util.logging or Logstash) is easily possible.\n7.3 QR-Code-Integration # For physical media, such as posters, packaging, or invitations, displaying a short link as a QR code is a useful extension. Integrating QR code generation into the URL shortener enables the direct generation of a visually encoded image of the link from the API.\nSince no external libraries are used, QR code generation can be performed using a compact Java-based algorithm, such as one based on bit matrix generation and SVG output. Alternatively, a Base64-encoded PNG file can be delivered via an endpoint URL such as /qr/{alias}. The underlying data structure remains unchanged – only the representation is extended.\nThis feature not only enhances practical utility but also expands the service\u0026rsquo;s reach across multiple media channels.\n7.4 Integration into messaging or tracking systems # In production architectures, a URL shortener typically operates in conjunction with other components. Instead, it is part of larger pipelines – for example, in email delivery, chatbots, content management systems, or user interaction tracking. Flexible integration with messaging systems such as Kafka, RabbitMQ, or simple webhooks allows every link creation or access to be transmitted as an event to external systems.\nIn a pure Java environment, this can be done via simple HTTP requests, log files, or asynchronous event queues. Scenarios are conceivable in which a notification is automatically sent to a third-party system for each new short link, for example, to generate personalised campaigns or for auditing purposes. Access to short links can also be mapped via events, which are subsequently statistically evaluated or visualised in dashboards.\nDepending on the level of integration, it is recommended to implement a dedicated event dispatcher that encapsulates incoming or outgoing events and forwards them in a loosely coupled manner. This keeps the shortener itself lean and responsibilities clearly distributed.\n8. Legal and ethical aspects # 8.1 Data protection for link tracking # A URL shortener that logs visits automatically operates within the framework of data protection law. As soon as data such as IP addresses, timestamps, or user agents are stored, it is considered personal information in the legal sense, at least potentially. In the European Union, such data falls under the General Data Protection Regulation (GDPR), which entails specific obligations for operators.\nThe technical capability for analytics—for example, through access counting or geo-IP analysis—should therefore not be enabled implicitly. Instead, a URL shortener should be designed so that tracking mechanisms must be explicitly enabled, ideally with clear labelling for the end user. A differentiated configuration that distinguishes between anonymised and personal data collection is strongly recommended in professional environments.\nAdditionally, when storing personal data, a record of processing activities must be maintained, a legal basis (e.g., legitimate interest or consent) must be specified, and a defined retention period must be established. For publicly accessible shorteners, this may mean that tracking remains deactivated by default or is controlled via consent mechanisms. The implementation of such control structures is not part of the core functionality, but is an integral part of data protection-compliant operations.\n8.2 Responsibility for forwarding # Another key point is the service provider\u0026rsquo;s responsibility for the content to which the link is redirected. Even if a shortener technically only implements a redirect, legal responsibility arises as soon as the impression arises that the operator endorses or controls the target content. This is especially true for public or embedded shorteners, such as those found in corporate portals or social platforms.\nThe challenge lies in distinguishing between technical neutrality and de facto mediation. It is therefore advisable to integrate legal protection mechanisms into the architecture, for example, through a policy that excludes the upload of specific domains, regular URL revalidation, or the use of abuse detection systems. In the event of misuse or complaints, immediate deactivation of individual mappings should be possible, ideally via a separate administration interface.\nThis responsibility is not only legally relevant but also has a reputational impact: Shorteners used to spread harmful content quickly lose their credibility – and possibly also their access to platforms or search engines.\n8.3 Transparency and disclosure of the destination address # A common criticism of URL shorteners is that the destination address is no longer visible to the user. This limits the ability to evaluate whether a link is trustworthy before clicking on it. From an ethical perspective, this raises the question of whether a shortener should offer a pre-check option.\nTechnically, this can be achieved through a special preview mode, such as via an appendage, by explicitly calling an API or HTML preview page that transparently resolves the mapping, for example, a link like https://short.ly/abc123+. Instead of redirecting immediately, the user first displays an information page that displays the original URL and redirects to the page if desired. This function can be supplemented with information about validity, access statistics, or trustworthiness.\nA transparent approach to redirects not only increases user acceptance but also reduces the potential for abuse, especially among security-conscious target groups. In sensitive environments, a mandatory preview page – for example, for all non-authenticated users – can be a helpful measure.\n9. Conclusion and outlook # 9.1 Lessons Learned # The development of a URL shortener in pure Java, without frameworks or external libraries, has demonstrated how even seemingly trivial web services, upon closer inspection, reveal themselves to be complex systems with diverse requirements. From the basic function of address shortening to security aspects and operational and legal implications, the result is a system that must be architecturally well-structured, yet flexible and extensible.\nThe importance of a clear separation of responsibilities is particularly important: A stable mapping store, a deterministic encoder, a secure yet straightforward REST API, and understandable error handling form the backbone of a robust service. Modern language tools from Java 24, such as records, sealed types, and virtual threads, enable a remarkably compact, type-safe, and concurrency-capable implementation.\nThe conscious decision against frameworks not only maximised the learning effect but also contributed to a deeper understanding of HTTP, data storage, thread safety, and API design – a valuable perspective for developers who want to operate in a technology-independent environment.\n9.2 Possible further developments (e.g. blockchain, DNSSEC) # Despite their apparent simplicity, URL shorteners represent a fascinating field for technological innovation. There are efforts to move away from centralised management of the mapping between short code and target URL, instead using decentralised technologies such as blockchain. In this case, each link is stored as a transaction, providing resistance to manipulation and historical traceability. In practice, however, this places high demands on latency and infrastructure, which is why such approaches have been used so far rarely in production.\nAnother development strand lies in integration with DNSSEC-based procedures. This not only signs the shortcode itself, but also cryptographically verifies the authenticity of the resolved host. This could combine trust and verification, especially in security-critical areas such as government services, banks, or certificate authorities.\nAI-supported heuristics, such as those for misuse detection or memory cleanup prioritisation, also offer potential. However, the integration of such mechanisms requires a data-efficient, explainable design that is compatible with applicable data protection regimes.\n9.3 Importance of URL shorteners in the context of digital sovereignty # In today\u0026rsquo;s digital landscape, URL shorteners are more than just a convenience feature; they are a valuable tool. They influence the visibility, accessibility, and traceability of content. The question of whether and how a link is modified or redirected has a direct impact on information sovereignty and transparency, and thus on digital sovereignty.\nEspecially in the public sector, educational institutions, or organisations with strict compliance requirements, URL shorteners should not be operated as outsourced cloud services; instead, they should be developed in-house or at least integrated in a controlled manner. A self-hosted solution not only allows complete control over data flows and access histories but also protects against censorship-like outages or data-driven tracking by third parties.\nThis makes the URL shortener, as inconspicuous as its function may seem, a strategic component of a trustworthy IT infrastructure. It exemplifies the question: Who controls the path of information? In this respect, a custom shortener is not just a tool, but also a statement of identity.\nThe next part will be about the implementation itself..\nHappy Coding\n","date":"10 June 2025","externalUrl":null,"permalink":"/posts/short-links-clear-architecture-a-url-shortener-in-core-java/","section":"Posts","summary":"A URL shortener seems harmless – but if implemented incorrectly, it opens the door to phishing, enumeration, and data leakage. In this first part, I’ll explore the theoretical and security-relevant fundamentals of a URL shortener in Java – without any frameworks, but with a focus on entropy, collision tolerance, rate limiting, validity logic, and digital responsibility. The second part covers the complete implementation: modular, transparent, and as secure as possible.\n","title":"Short links, clear architecture – A URL shortener in Core Java","type":"posts"},{"content":"A deep look into Java’s HashMap traps – visually demonstrated with Vaadin Flow.\nThe silent danger in the standard library # The use of HashMap and HashSet is a common practice in everyday Java development. These data structures offer excellent performance for lookup and insert operations, as long as their fundamental assumptions are met. One of them is hashCode() of a key remains stable. But what if that\u0026rsquo;s not the case?\nThis is precisely where one of the most subtle and dangerous traps of the Java standard library lurks: mutable key objects. In this article, we not only demonstrate why this constellation is problematic but also illustrate the phenomenon interactively using Vaadin Flow. Readers will learn how the HashMap works internally, why equals() alone is not enough, and how to use modern language tools to generate robust, immutable keys.\nThe silent danger in the standard library The fundamental problem: identity, hash codes and lookup The classic mistake hashCode() depends on variable attributes Security-critical consequences: When loss of consistency becomes an attack surface Interactive Demo with Vaadin Flow HashSet Strategies to avoid Why are other map implementations not affected Conclusion: The price of convenience Demo source code to try it out yourself The fundamental problem: identity, hash codes and lookup # Internally, each HashMap is an array of buckets, where the hash code of the key determines the position of an entry. The insertion process (put(K key, V value)) looks like this: First, the map calls key.hashCode() and calculates the index in the bucket array from this value, using internal spreading. If entries already exist at this position (e.g., due to hash collisions), a linear search is performed within the bucket or – if there are enough collisions – i.e., if there are more than eight entries in a bucket and the underlying array exceeds a specific size (default value: 64) – the bucket is internally transformed from a simple linked list into a balanced binary tree structure (more precisely: a red-black tree). This conversion improves the lookup performance from linear time O(n) to logarithmic time O(log n). The decision for these thresholds is based on the observation that collisions are rare in well-chosen hash functions, and the overhead of a tree structure is only worthwhile with a high entry density. equals() is used to identify the appropriate key or create a new entry.\nWhen accessing (get(Object key)) the same process occurs: The hash code of the transferred key is calculated again, the corresponding bucket is determined and searched for a matching key via equals() However, if the hash code of the object differs from the original put() have changed, a different bucket is addressed – the original entry then remains invisible.\nRemoving (remove(Object key)) is subject to the same mechanisms: the map searches for the correct bucket via hashCode() and then compares the keys using equals(). Consistency of the hash-relevant data over the entire lifetime of the key is therefore essential. Only if this is guaranteed can the HashMap ensure its efficiency and correctness.\nThis means: Even access to the correct bucket depends exclusively on the result of the method hashCode() This value is calculated immediately upon access, combined with an internal transformation (such as bitwise shifting and XOR operations to achieve a more even distribution across the bucket array), and then used to index the bucket array. If the return value of hashCode(). After inserting an object into the map, for example, by mutating an attribute that is included in the calculation, the newly calculated index points to a different bucket. However, the object being searched for does not exist there, which is why the map can no longer find the entry. This behaviour is not a malfunction of the HashMap, but the direct consequence of a breach of the fundamental contract hashCode() must remain consistent while maintaining the same internal object state. Violating this contract is abusing the HashMap in a way it was not designed for, with potentially serious consequences for data integrity and program logic.\nThe classic mistake hashCode() depends on variable attributes # Let\u0026rsquo;s take the following example: An instance of the class Person with the values name = \u0026ldquo;Alice\u0026rdquo; and id = 42 is created and stored as a key in a HashMap At the time of insertion, the map calculates the hashCode() based on the current state of the object – i.e. Objects.hash(\u0026ldquo;Alice\u0026rdquo;, 42) – and saves the entry in the corresponding bucket. After that, the field name of the object, e.g., is set to the value \u0026ldquo;Bob\u0026rdquo;. This changes the return value of hashCode() , for example, Objects.hash(\u0026ldquo;Bob\u0026rdquo;, 42) , which addresses a different bucket.\nIf you now try to access the map again with the same object, map.get(originalPerson) – the operation fails. The map calculates a new index from the current hash code, looks in the corresponding bucket, and doesn\u0026rsquo;t find a matching entry. The original object is technically still in the map, but cannot be found.\nThe situation becomes even more misleading when one considers theentrySet() method**.** The entry is visible there, since the iteration is carried out directly via the internal chains, independent of the hash code. A comparison via equals() works with a newly created, identical object – after all, equals() is typically based on content equality and not on memory address or hash code.\nThis makes the HashMap de facto inconsistent: The element is still physically stored in the internal data array, but cannot be accessed via regular access paths, such as get(key), can no longer be found. To demonstrate this effect reproducibly, consider the following code example:\nimport java.util.*; class Person { String name; you hand; Person(String name, int id) { this.name = name; this.id = id; } @Override public boolean equals(Object o) { return o instanceof Person p \u0026amp;\u0026amp; Objects.equals(name, p.name) \u0026amp;\u0026amp; id == p.id; } @Override public int hashCode() { return Objects.hash(name, id); } } public class MutableHashDemo { public static void main(String[] args) { Person p = new Person(\u0026#34;Alice\u0026#34;, 42); Map\u0026lt;Person, String\u0026gt; map = new HashMap\u0026lt;\u0026gt;(); map.put(p, \u0026#34;Value\u0026#34;); System.out.println(\u0026#34;Before change:\u0026#34;); System.out.println(\u0026#34;map.get(p): \u0026#34; + map.get(p)); // Mutation of the key p.name = \u0026#34;Bob\u0026#34;; System.out.println(\u0026#34;After change:\u0026#34;); System.out.println(\u0026#34;map.get(p): \u0026#34; + map.get(p)); System.out.println(\u0026#34;Enthält key via entrySet: \u0026#34; + map.entrySet().stream() .anyMatch(e -\u0026gt; e.getKey().equals(p))); } } The demo source code is available on github athttps://github.com/svenruppert/Blog\u0026mdash;Core-Java\u0026mdash;Mutable-HashMap-Keys-in-Java/blob/main/src/test/java/junit/com/svenruppert/MutableHashCodeDemoTest.java\nThis example demonstrates that map.get(p) returns null after the change, although the entrySet() still contains the entry. The reason hashCode() returns a different value after the mutation so that the original bucket is no longer addressed. equals() alone does not help in this case, since the lookup fails due to the wrong index.\nSecurity-critical consequences: When loss of consistency becomes an attack surface # While the loss of referentiality in a HashMap may appear at first glance to be merely a technical issue, closer inspection reveals security-relevant implications, particularly in systems that utilise state caching, authentication, or access control. If an object serves as a key in security-critical maps or sets – e.g., for detecting active sessions, auth tokens, or user permissions – and its hash code subsequently changes, a logical error condition arises. The system no longer \u0026ldquo;sees\u0026rdquo; the object, even though it still exists. This can lead to unintended access gaps or, worse still, unattended persistence of state.\nA particularly dangerous scenario arises when mutable keys can be influenced from outside. One conceivable scenario is where an attacker can inject controlled values ​​into an object that then serves as a key. As soon as this key is changed—for example, through manipulated API usage or faulty deserialisation— it is removed from access control, even though it technically still exists. The result: logical access without valid authorisation.\nIn extreme cases, this can even lead to typical security-critical patterns, such as:\nAuthorisation Bypass: An object is modified before the test, the contains() fails, and access is granted incorrectly.\nResource Lock Hijack: A lock object is created via a HashSet orMap managed by the system, but it can no longer be removed after mutation**–** a deadlock or race condition threatens.\nDenial of service due to hash collisions : If a system stores (or mutates) many mutable objects with controlled hash codes, hash collisions can be deliberately triggered, and performance problems can be created.\nAlthough a classic buffer overflow doesn\u0026rsquo;t occur directly in Java due to the memory safety of the JVM model, the structural effect of an inconsistent HashMap is similar to an overflow at the logical level: An access lands in the \u0026ldquo;wrong memory area\u0026rdquo; (bucket), and the object is present but functionally invisible. This is a key point for security auditors and architects: Loss of consistency in hash-based structures can not only lead to incorrect behaviour but also become a potential entry point for complex attacks.\nInteractive Demo with Vaadin Flow # To make the described problem tangible, a minimalist demonstration application is recommended, e.g., with Vaadin Flow. The demo aims to allow the user to observe live how an object in a HashMap becomes “invisible” after a mutation.\nThe application consists of a simple Vaadin view with the following UI elements:\nInput fields for name and ID A button to insert an object into a HashMap A button to modify the name (and thus the hash code) A button to execute map.get() A button for iterating over entrySet() The view constructor first creates the graphical user interface. Two input fields – one for the name, one for the ID – are used to interact with the Person -Object.\nprivate final TextField nameField = new TextField(\u0026#34;Name\u0026#34;); private final TextField idField = new TextField(\u0026#34;ID\u0026#34;); private final TextArea output = new TextArea(\u0026#34;Output\u0026#34;); private final Person mutableKey = new Person(\u0026#34;Alice\u0026#34;, 42); private final Map\u0026lt;Person, String\u0026gt; map = new HashMap\u0026lt;\u0026gt;(); Several buttons allow you to perform specific operations on the Map. The \u0026ldquo;put(key, value) \u0026quot; button adds the current mutable Key-Object as key with the value \u0026ldquo;Saved \u0026quot; to the map. This uses exactly the object whose name and id were determined at the beginning – here \u0026ldquo;Alice\u0026rdquo; and 42.\nButton putButton = new Button(\u0026#34;put(key, value)\u0026#34;, _ -\u0026gt; { map.put(mutableKey, \u0026#34;Saved\u0026#34;); }); About the button \u0026ldquo;Change name\u0026rdquo;, the name of the mutableKey object at runtime. This leads to a state in which the contents of the object—and thus its hash code—change after it is inserted into the map.\nButton mutateButton = new Button(\u0026#34;Change name\u0026#34;, _ -\u0026gt; { mutableKey.name = nameField.getValue(); }); The \u0026ldquo;get(key)\u0026rdquo; Button demonstrates this effect: The method map.get(mutableKey) attempts to retrieve the value using the (modified) object as a key.\nButton getButton = new Button(\u0026#34;get(key)\u0026#34;, and -\u0026gt; { String result = map.get(mutableKey); output.setValue(\u0026#34;Result of get(): \u0026#34; + result); }); To illustrate this effect and, at the same time, provide an alternative access, the button \u0026ldquo;search entrySet()\u0026rdquo; has been added. It manually iterates through all key-value pairs of the map using the Stream API and compares the keys via equals() , regardless of the internal bucket structure. This means that an entry with the changed key object can still be found, provided equals() is correctly implemented and independent of the hashCode function.\nButton iterateButton = new Button(\u0026#34;search entrySet()\u0026#34;, and -\u0026gt; { String result = map.entrySet().stream() .filter(entry -\u0026gt; entry.getKey().equals(mutableKey)) .map(Map.Entry::getValue) .findFirst() .orElse(\u0026#34;Not found via equals()\u0026#34;); output.setValue(\u0026#34;entrySet(): \u0026#34; + result); }); //Here is the complete source code of the view. @Route(value = PATH, layout = MainLayout.class) public class VersionOneView extends VerticalLayout implements HasLogger { public static final String PATH = \u0026#34;versionone\u0026#34;; private final TextField nameField = new TextField(\u0026#34;Name\u0026#34;); private final TextField idField = new TextField(\u0026#34;ID\u0026#34;); private final TextArea output = new TextArea(\u0026#34;Output\u0026#34;); private final Person mutableKey = new Person(\u0026#34;Alice\u0026#34;, 42); private final Map\u0026lt;Person, String\u0026gt; map = new HashMap\u0026lt;\u0026gt;(); public VersionOneView() { logger().info(\u0026#34;Initializing VersionOneView\u0026#34;); nameField.setValue(mutableKey.getName()); idField.setValue(String.valueOf(mutableKey.getId())); output.setWidth(\u0026#34;600px\u0026#34;); Button putButton = new Button(\u0026#34;put(key, value)\u0026#34;, _ -\u0026gt; { logger().info(\u0026#34;Putting value into map with key: {}\u0026#34;, mutableKey); map.put(mutableKey, \u0026#34;Saved\u0026#34;); output.setValue(\u0026#34;Inserted: \u0026#34;+ mutableKey); }); Button mutateButton = new Button(\u0026#34;Change name\u0026#34;, _ -\u0026gt; { logger().info(\u0026#34;Changing name from {} to {}\u0026#34;, mutableKey.getName(), nameField.getValue()); mutableKey.setName(nameField.getValue()); output.setValue(\u0026#34;Name changed to: \u0026#34;+ mutableKey.getName()); }); Button getButton = new Button(\u0026#34;get(key)\u0026#34;, and -\u0026gt; { logger().info(\u0026#34;Getting value for key: {}\u0026#34;, mutableKey); String result = map.get(mutableKey); logger().info(\u0026#34;Get result: {}\u0026#34;, result); output.setValue(\u0026#34;Result of get(): \u0026#34; + result); }); Button iterateButton = new Button(\u0026#34;search entrySet()\u0026#34;, and -\u0026gt; { logger().info(\u0026#34;Searching through entrySet for key: {}\u0026#34;, mutableKey); String result = map.entrySet().stream() .filter(entry -\u0026gt; entry.getKey().equals(mutableKey)) .map(Map.Entry::getValue) .findFirst() .orElse(\u0026#34;Not found via equals()\u0026#34;); logger().info(\u0026#34;EntrySet search result: {}\u0026#34;, result); output.setValue(\u0026#34;entrySet(): \u0026#34; + result); }); add(nameField, idField, putButton, mutateButton, getButton, iterateButton, output); logger().info(\u0026#34;VersionOneView initialization completed\u0026#34;); } public class Person { String name; int id; public Person(String name, int id) { this.name = name; this.id = id; } //SNIP getter setter @Override public boolean equals(Object o) { return theinstanceof Person p \u0026amp;\u0026amp; Objects.equals(name, p.name) \u0026amp;\u0026amp; id == p.id; } @Override public int hashCode() { return Objects.hash(name, id); } @Override public String toString() { return name + \u0026#34; (\u0026#34; + id + \u0026#34;)\u0026#34;; } } This concise yet powerful view enables any reader to directly observe the effects of changing keys.\nThe demo source code is available on github athttps://github.com/svenruppert/Blog\u0026mdash;Core-Java\u0026mdash;Mutable-HashMap-Keys-in-Java/blob/main/src/main/java/com/svenruppert/flow/views/version01/VersionOneView.java** **\nNow, let\u0026rsquo;s modify the view slightly and display all entries from the EntrySet. The hash codes are compared, and the result is output.\nButton iterateButton = new Button(\u0026#34;search entrySet()\u0026#34;, _ -\u0026gt; { logger().info(\u0026#34;Searching through entrySet for key: {}\u0026#34;, mutableKey); StringBuilder result = new StringBuilder(); map.forEach((key, value) -\u0026gt; { boolean isMatch = key.equals(mutableKey); result.append(String.format(\u0026#34;\u0026#34;\u0026#34; Key: %s, Value: %s, HashCode %s Match: %s\u0026#34;\u0026#34;\u0026#34;, key, value, key.hashCode(), isMatch ? \u0026#34;And\u0026#34; : \u0026#34;No\u0026#34;)); result.append(\u0026#34;\\n\u0026#34;); }); logger().info(\u0026#34;EntrySet search result: {}\u0026#34;, result); output.setValue(\u0026#34;entrySet():\\n\u0026#34; + (!result.isEmpty() ? result.toString() : \u0026#34;Map is empty\u0026#34;)); }); Now you can clearly see how an instance now exists multiple times in the same hash map. If you think about it, you can think of some very unpleasant applications that could even be used to specifically attack a system. But more on that in another blog post.\nHashSet # Since a HashSet internally is nothing other than a HashMap , where the keys are the actual set elements and the values ​​are a constant dummy (such as PRESENT = new Object()), all the problems described here apply in the same way. If an object is modified after being inserted into the set, and this change affects one of the attributes that is used in the calculation of hashCode() , the object is searched in the wrong bucket. The method contains() returnsfalse, even though the object is stored in the set. remove() fails because the same logic as get() takes effect: the HashMap finds the bucket using the current hash code and does not recognise the key there.\nThis behaviour is particularly critical when HashSet is used to temporarily store security-relevant information, for example, to manage already authenticated users, valid tokens, or temporary permissions. An unintentionally mutated key can lead to access being incorrectly denied or even security checks being bypassed—a classic case of a logical security vulnerability that is barely noticeable and difficult to debug.\nIn safety-critical modules, set elements should therefore always be immutable and defensively constructed. This means, in concrete terms, no changeable fields, no public modifiability, and – ideally – construction via a record or builder with final attributes.\nSet\u0026lt;Person\u0026gt; people = new HashSet\u0026lt;\u0026gt;(); people.add(p); // p.hashCode() ist X p.name = \u0026#34;Malicious\u0026#34;; System.out.println(people.contains(p)); // false Strategies to avoid # The simplest and most effective strategy is to use only immutable objects as keys. Since Java 16, records are the idiomatically correct tool for this. They automatically generate equals() and hashCode() based on final fields.\nrecord PersonKey(String name, int id) {}\nAlternatively, if mutable state is unavoidable, the object before A modification can remove the hash code from the map and then reinsert it with an updated hash code. This is a dangerous workaround that is rarely implemented correctly in practice.\nWhy are other map implementations not affected # The problem described here occurs specifically in HashMap and structures based on it, such as HashSet, because a calculated hash code determines the access path. Other map implementations in the JDK – such as TreeMap , LinkedHashMap , or EnumMap - are not affected in this respect or only to a lesser extent.\nThe TreeMap, for example, is not based on hash code-based indexing, but on a sorted tree structure (red-black tree). It uses either the natural order of the keys (via Comparable) or a provided Comparator. This means that access does not depend on the result of hashCode() but from the stable comparability through compareTo() or compare(K1, K2). A change to an attribute that is included in the comparison logic can also lead to inconsistent behaviour, for example, get() or remove(). However, the mechanism is more transparent and controllable because sorting is done using a clearly defined comparison function.\nAlso, LinkedHashMap , although internally a HashMap , is not immune to the mutable hashcode problem because it uses the same bucket logic. However, it also provides a deterministic iteration order, which can aid in debugging, but does not alter the underlying problem.\nThe EnumMap. Finally, it is protected against this problem by design, as it is exclusively of enum types as keys. These are immutable by language definition and have stable, final hash codes. Thus, mutation is impossible, and the map remains robust against this class of errors.\nYou should therefore carefully consider which map implementation is used in each context, and whether a stable key state can be guaranteed or whether alternative ordering mechanisms should be used.\nConclusion: The price of convenience # In daily development, the problem is often overlooked because it only manifests itself in specific situations – for example, after multiple operations, in multi-threading, or integrations across API boundaries. However, the consequences are severe: lost entries, unexplained null values , and inconsistent state.\nAnyone who wants to use HashMaps efficiently and correctly should respect their internal rules, particularly ensuring that key objects do not change subsequently.\nDemo source code to try it out yourself # The complete source code for the demo view with Vaadin Flow is publicly available on GitHub. The application can be run efficiently locally using\u0026rsquo; mvn jetty: run\u0026rsquo;. The web application will then be available at the address http://localhost:8080/.\nGitHub: https://github.com/svenruppert/Blog---Core-Java---Mutable-HashMap-Keys-in-Java\nHappy Coding\nSven\n","date":"6 June 2025","externalUrl":null,"permalink":"/posts/if-hashcode-lies-and-equals-is-helpless/","section":"Posts","summary":"A deep look into Java’s HashMap traps – visually demonstrated with Vaadin Flow.\nThe silent danger in the standard library # The use of HashMap and HashSet is a common practice in everyday Java development. These data structures offer excellent performance for lookup and insert operations, as long as their fundamental assumptions are met. One of them is hashCode() of a key remains stable. But what if that’s not the case?\n","title":"If hashCode() lies and equals() is helpless","type":"posts"},{"content":"Vaadin Flow is a robust framework for building modern web applications in Java, where all UI logic is implemented on the server side. In this blog post, we\u0026rsquo;ll make a simple file management application step by step that allows users to upload files, save them to the server, and download them again when needed. This is a great way to demonstrate how to build protection against CWE-22, CWE-377, and CWE-778 step by step.\nBasic project structure Add file upload functionality. Add download functionality Best Practices uses Java NIO. CWE-22: Path Traversal and Protection Measures CWE-377: Insecure temporary files CWE-778: Insufficient logging Summary In this example, we\u0026rsquo;re focusing exclusively on functionality rather than on graphic design. The latter has been intentionally kept very simple to focus on the technical aspects.\nThe source texts for this article can be found at:https://github.com/Java-Publications/Blog\u0026mdash;Secure-Coding-Practices\u0026mdash;CWE-022\u0026ndash;377\u0026ndash;778\u0026mdash;A-practical-Demo** __**\nBasic project structure # To begin, we will create a new Vaadin project. The easiest way to do this is via the project starter, which you can find at https://start.vaadin.com/ or by using an existing Maven template. The file structure of our project essentially looks like this:\nScreenshot\nThe file **MainView.java** will be the application\u0026rsquo;s central entry point. Here, we will implement the user interface and the logic for file uploads and downloads.\nAdd file upload functionality. # First, we will create a simple user interface that allows users to upload files. In **MainView.java** , our basic setup looks like this:\npackage com.svenruppert.filemanager; import com.vaadin.flow.component.button.Button; import com.vaadin.flow.component.notification.Notification; import com.vaadin.flow.component.orderedlayout.VerticalLayout; import com.vaadin.flow.component.upload.Upload; import com.vaadin.flow.component.upload.receivers.MemoryBuffer; import com.vaadin.flow.router.Route; import java.io.File; import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; @Route public class MainView extends VerticalLayout { public MainView() { MemoryBuffer buffer = new MemoryBuffer(); Upload upload = new Upload(buffer); upload.setMaxFiles(1); upload.addSucceededListener(event -\u0026gt; { String fileName = event.getFileName(); try (InputStream inputStream = buffer.getInputStream()) { File targetFile = new File(\u0026#34;uploads/\u0026#34; + fileName); targetFile.getParentFile().mkdirs(); try (FileOutputStream outputStream = new FileOutputStream(targetFile)) { inputStream.transferTo(outputStream); } Notification.show(\u0026#34;File \u0026#34; + fileName + \u0026#34; uploaded successfully!\u0026#34;); } catch (IOException e) { Notification.show(\u0026#34;Error uploading file\u0026#34;); } }); add(upload); } } In this code, we use MemoryBuffer to temporarily save the uploaded file and then write it to the uploads/ directory. If the target directory doesn\u0026rsquo;t already exist, it will be created automatically. Using **MemoryBuffer** allows easy and secure file management before it is written to the hard disk.\nAdd download functionality # To list the downloadable files, we\u0026rsquo;ll add a button that allows users to download any file from the directory. This improves the user experience and ensures users can access their uploaded files easily. Here we\u0026rsquo;ll extend the user interface:\nimport com.vaadin.flow.component.grid.Grid; import com.vaadin.flow.component.html.Anchor; import java.io.File; public class MainView extends VerticalLayout { public MainView() { // Upload function as described above MemoryBuffer buffer = new MemoryBuffer(); Upload upload = new Upload(buffer); upload.setMaxFiles(1); upload.addSucceededListener(event -\u0026gt; { String fileName = event.getFileName(); try (InputStream inputStream = buffer.getInputStream()) { File targetFile = new File(\u0026#34;uploads/\u0026#34; + fileName); targetFile.getParentFile().mkdirs(); try (FileOutputStream outputStream = new FileOutputStream(targetFile)) { inputStream.transferTo(outputStream); } Notification.show(\u0026#34;File \u0026#34; + fileName + \u0026#34; uploaded successfully!\u0026#34;); updateFileList(); } catch (IOException e) { Notification.show(\u0026#34;Error uploading file\u0026#34;); } }); add(upload); updateFileList(); } private void updateFileList() { removeAll(); File folder = new File(\u0026#34;uploads\u0026#34;); File[] listOfFiles = folder.listFiles(); if (listOfFiles != null) { for (File file : listOfFiles) { if (file.isFile()) { Anchor downloadLink = new Anchor(\u0026#34;/uploads/\u0026#34; + file.getName(), file.getName()); downloadLink.getElement().setAttribute(\u0026#34;download\u0026#34;, true); add(downloadLink); } } } } } Using the method **updateFileList()** displays the files stored in the uploads/ directory as a list, and creates an anchor element for each file, which serves as a download link. This makes the interface more intuitive and allows users to manage uploaded files easily.\nBest Practices uses Java NIO. # We can use Java NIO (New Input/Output) instead of traditional IO streams to improve efficiency and security when handling files. Java NIO provides non-blocking IO operations, enabling better performance and scalability. It also supports more flexible and secure file system operations.\nWe adapt our code to use Java NIO classes like **Files** and **Path** to save the uploaded files. Here\u0026rsquo;s an improved version of the upload code that uses Java NIO:\nimport java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardCopyOption; public class MainView extends VerticalLayout { public MainView() { MemoryBuffer buffer = new MemoryBuffer(); Upload upload = new Upload(buffer); upload.setMaxFiles(1); upload.addSucceededListener(event -\u0026gt; { String fileName = event.getFileName(); Path targetPath = Paths.get(\u0026#34;uploads\u0026#34;).resolve(fileName); try (InputStream inputStream = buffer.getInputStream()) { Files.createDirectories(targetPath.getParent()); Files.copy(inputStream, targetPath, StandardCopyOption.REPLACE_EXISTING); Notification.show(\u0026#34;File \u0026#34; + fileName + \u0026#34; uploaded successfully!\u0026#34;); updateFileList(); } catch (IOException e) { Notification.show(\u0026#34;Error uploading file\u0026#34;); } }); add(upload); updateFileList(); } } This version uses the Files and Paths classes to create directories and store files. This improves code readability and maintainability and leverages the advantages of the NIO classes, such as better exception handling and more flexible path operations. Using Java NIO makes file operations more efficient and secure, especially when working with multiple threads concurrently.\nCWE-22: Path Traversal and Protection Measures # CWE-22, also known as \u0026ldquo;Path Traversal,\u0026rdquo; is a vulnerability that occurs when users can access unauthorised files and directories in the file system via insecure path names. This is typically done by users inserting special character strings such as ../ into filenames to extend beyond the boundaries of the permitted directory. If this is not controlled correctly, an attacker could gain access to critical system files, potentially leading to a serious security compromise.\nIn our file management application, path traversal is risky if the filename is used directly and without verification to save or retrieve a file in the file system. Attackers could attempt to manipulate path information and overwrite or read files outside the intended directory.\nTo avoid vulnerabilities like CWE-22 (Path Traversal) in your application, you should be especially careful with user-supplied filenames. Attackers could attempt to access files outside their intended location using manipulated path names—for example, by entering something like../../etc/passwd. Therefore, checking and cleaning all paths and file names is essential before using them. Here are a few best practices you should implement:\nPath cleanup: Use methods like Path.normalize()from the Java NIO package to clean paths. This automatically removes dangerous constructs such as double slashes or relative elements (..) that attackers could use to traverse directories.\nDirectory restriction: Make sure that the path you use to save files is within a secure, predefined home directory – e.g. uploads/ You can verify this by comparing the cleaned destination path with the expected base directory. You should allow the upload only if the destination path is truly within the allowed range.\nFile name validation: It\u0026rsquo;s not enough to check the path alone. The filename itself can also contain dangerous characters. It\u0026rsquo;s best to allow only simple, non-critical characters, such as letters, numbers, hyphens, and periods. Using a regular expression like[a-za-Z0-9._—], you can specifically remove or replace everything else.\nConsidering these points makes it significantly more difficult for attackers to control paths and protect your server structure and users\u0026rsquo; sensitive data.\nHere is a customised version of our application that implements protections against CWE-22:\nimport java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardCopyOption; public class MainView extends VerticalLayout { public MainView() { MemoryBuffer buffer = new MemoryBuffer(); Upload upload = new Upload(buffer); upload.setMaxFiles(1); upload.addSucceededListener(event -\u0026gt; { String fileName = sanitizeFileName(event.getFileName()); Path targetPath = Paths.get(\u0026#34;uploads\u0026#34;).resolve(fileName).normalize(); // Prevent the path from being outside the upload directory if (!targetPath.startsWith(Paths.get(\u0026#34;uploads\u0026#34;).toAbsolutePath())) { Notification.show(\u0026#34;Invalid file path! Upload aborted.\u0026#34;); return; } try (InputStream inputStream = buffer.getInputStream()) { Files.createDirectories(targetPath.getParent()); Files.copy(inputStream, targetPath, StandardCopyOption.REPLACE_EXISTING); Notification.show(\u0026#34;File \u0026#34; + fileName + \u0026#34; uploaded successfully!\u0026#34;); updateFileList(); } catch (IOException e) { Notification.show(\u0026#34;Error uploading file\u0026#34;); } }); add(upload); updateFileList(); } private String sanitizeFileName(String fileName) { return fileName.replaceAll(\u0026#34;[^a-zA-Z0-9._-]\u0026#34;, \u0026#34;_\u0026#34;); } } In this updated version of the application, we\u0026rsquo;ve added a sanitizeFileName() method to ensure the filename doesn\u0026rsquo;t contain any dangerous characters. We also normalise the path with Path.normalize() and verify that the final path is within the desired uploads directory. If the path is outside this directory, the upload is aborted and an appropriate error message is displayed.\nThese changes ensure that attackers cannot gain unauthorised access to the file system by manipulating file names. This keeps the application secure and protects both the server and the data stored on it from misuse.\nCWE-377: Insecure temporary files # CWE-377, also known as \u0026ldquo;Insecure Temporary File,\u0026rdquo; describes a security vulnerability that occurs when you create temporary files in a way that can be exploited by attackers. Such files are often used to cache content, either during processing or for temporary storage in general. However, if you create them insecurely, attackers could access, tamper with, or even overwrite them.\nA typical attack scenario: An attacker creates a symbolic link (symlink) pointing to a critical system file – your application then unknowingly writes data there. Or they manipulate the file while it\u0026rsquo;s still being processed, thereby introducing unwanted content or malicious code into the system. The consequences can range from data loss and integrity violations to a complete system compromise.\nTemporary files are also created in your file management application—for example, when processing uploads. Therefore, you must create these files securely. Here are a few best practices to effectively avoid CWE-377:\nSafe methods for creating temporary files: In Java, use the Files.createTempFile() method. It automatically creates a temporary file with a unique, random name, reducing the risk of attackers accessing it.\nRestrict access rights: Only your process can access the temporary file. Set the file permissions so no other users or services have access—this is especially important in shared environments.\nUse unpredictable file names: Avoid using fixed or easily guessed names for temporary files. Otherwise, attackers could create a file with the same name beforehand or overwrite existing ones.\nConsidering these points, you can securely handle temporary files and protect your application from an often underestimated attack vector.\nimport java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardCopyOption; public class MainView extends VerticalLayout { public MainView() { MemoryBuffer buffer = new MemoryBuffer(); Upload upload = new Upload(buffer); upload.setMaxFiles(1); upload.addSucceededListener(event -\u0026gt; { String fileName = sanitizeFileName(event.getFileName()); Path targetPath = Paths.get(\u0026#34;uploads\u0026#34;).resolve(fileName).normalize(); // Prevent the path from being outside the upload directory if (!targetPath.startsWith(Paths.get(\u0026#34;uploads\u0026#34;).toAbsolutePath())) { Notification.show(\u0026#34;Invalid file path! Upload aborted.\u0026#34;); return; } try (InputStream inputStream = buffer.getInputStream()) { // Create a secure temporary file Path tempFile = Files.createTempFile(\u0026#34;upload_\u0026#34;, \u0026#34;tmp\u0026#34;); Files.copy(inputStream, tempFile, StandardCopyOption.REPLACE_EXISTING); // Move the temporary file to the target directory Files.createDirectories(targetPath.getParent()); Files.move(tempFile, targetPath, StandardCopyOption.REPLACE_EXISTING); Notification.show(\u0026#34;File \u0026#34; + fileName + \u0026#34; uploaded successfully!\u0026#34;); updateFileList(); } catch (IOException e) { Notification.show(\u0026#34;Error uploading file\u0026#34;); } }); add(upload); updateFileList(); } private String sanitizeFileName(String fileName) { return fileName.replaceAll(\u0026#34;[^a-zA-Z0-9._-]\u0026#34;, \u0026#34;_\u0026#34;); } } In this revised version, you create a secure temporary file with Files.createTempFile(). You use this to temporarily store the uploaded content before moving it to the final destination directory, ensuring that third parties cannot tamper with the temporary file.\nOnly after the content has been successfully saved do you move the file to the correct location in the file system. This way, you can upload the files in a controlled and secure manner and minimise the risk of temporary files becoming a gateway for attackers.\nCWE-778: Insufficient logging # CWE-778, also known as \u0026ldquo;Insufficient Logging,\u0026rdquo; describes a security vulnerability that occurs when your application doesn\u0026rsquo;t log enough to detect and track security-relevant events. If you don\u0026rsquo;t maintain detailed logging, attempted attacks, unauthorised access, or system errors may go unnoticed for a long time, or you may not have enough information to respond appropriately in an emergency.\nEspecially in safety-critical applications, you should document all critical actions and errors to understand what happened later clearly. This is the only way to allow yourself or your team to respond quickly to incidents and learn from them.\nIf you log too little or nothing at all, you may miss signs of attacks, such as suspicious file names (path traversal), repeated failed uploads, or unauthorised access. To prevent this, you should keep a few basic things in mind:\nLog security-relevant events: Log all critical actions, such as file uploads, file accesses, path checks, or rejected requests, along with timestamps and context.\nRecord errors and exceptions: Make sure that you display a message for all errors and exceptions and record the exact cause in the log. This allows you to search for specific sources of errors later.\nUse a central logging solution: Use proven logging frameworks like SLF4J in combination with Logback or Log4j2. This ensures that all log messages are recorded consistently, structured, and configurable for your environment.\nBelow is a customised version of the application, which allows you to insert log messages at security-critical points. This gives you essential insights during operation and will enable you to react quickly to problems.\nimport java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardCopyOption; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class MainView extends VerticalLayout { private static final Logger logger = LoggerFactory.getLogger(MainView.class); public MainView() { MemoryBuffer buffer = new MemoryBuffer(); Upload upload = new Upload(buffer); upload.setMaxFiles(1); upload.addSucceededListener(event -\u0026gt; { String fileName = sanitizeFileName(event.getFileName()); Path targetPath = Paths.get(\u0026#34;uploads\u0026#34;).resolve(fileName).normalize(); // Prevent the path from being outside the upload directory if (!targetPath.startsWith(Paths.get(\u0026#34;uploads\u0026#34;).toAbsolutePath())) { Notification.show(\u0026#34;Invalid file path! Upload aborted.\u0026#34;); logger.warn(\u0026#34;Invalid file path attempt: {}\u0026#34;, targetPath); return; } try (InputStream inputStream = buffer.getInputStream()) { // Create a secure temporary file Path tempFile = Files.createTempFile(\u0026#34;upload_\u0026#34;, \u0026#34;tmp\u0026#34;); Files.copy(inputStream, tempFile, StandardCopyOption.REPLACE_EXISTING); // Move the temporary file to the target directory Files.createDirectories(targetPath.getParent()); Files.move(tempFile, targetPath, StandardCopyOption.REPLACE_EXISTING); Notification.show(\u0026#34;File \u0026#34; + fileName + \u0026#34; uploaded successfully!\u0026#34;); logger.info(\u0026#34;File {} successfully uploaded to {}\u0026#34;, fileName, targetPath); updateFileList(); } catch (IOException e) { Notification.show(\u0026#34;Error uploading file\u0026#34;); logger.error(\u0026#34;Error uploading file {}: {}\u0026#34;, fileName, e.getMessage(), e); } }); add(upload); updateFileList(); } private void updateFileList() { removeAll(); File folder = new File(\u0026#34;uploads\u0026#34;); File[] listOfFiles = folder.listFiles(); if (listOfFiles != null) { for (File file : listOfFiles) { if (file.isFile()) { Anchor downloadLink = new Anchor(\u0026#34;/uploads/\u0026#34; + file.getName(), file.getName()); downloadLink.getElement().setAttribute(\u0026#34;download\u0026#34;, true); add(downloadLink); logger.info(\u0026#34;File {} added to download list\u0026#34;, file.getName()); } } } else { logger.warn(\u0026#34;Directory \u0026#39;uploads\u0026#39; could not be read or is empty.\u0026#34;); } } private String sanitizeFileName(String fileName) { return fileName.replaceAll(\u0026#34;[^a-zA-Z0-9._-]\u0026#34;, \u0026#34;_\u0026#34;); } } The revised implementation uses the SLF4J (Simple Logging Facade for Java) logging API in combination with a specific implementation such as Logback. This separation of API and logging engine offers flexibility in choosing the underlying technology and facilitates integration into different runtime environments. The goal is to systematically record security-relevant events to ensure both transparency and traceability of security-critical actions during operation.\nIn the context of security-conscious web applications—especially those that work with user-generated content such as file uploads—comprehensive and structured logging is essential. You should explicitly record the following event types:\nPath traversal tests (CWE-22): If a user attempts to access paths outside the intended memory area through manipulated input, you should report this with a WARN or higher log level. The logged information should include both the compromised path and the associated session or IP address to enable later correlation with other events.\nSuccessful file uploads: Every completed upload process should be documented in the log, including the cleaned-up file name, the target path in the file system, and optional context information such as the user ID or upload time. This allows for complete traceability and also serves as an audit trail.\nUpload error: If an exception occurs while saving the file (e.g., due to file system errors, access violations, or corrupted streams), you should log the full stack trace and all relevant metadata (file name, user context, time). This is essential for efficient error diagnosis and allows you to distinguish potential attack attempts from legitimate error cases.\nDynamic update of the file view: Logging can also be helpful for internal system actions, such as refreshing the list of available files to track when a file becomes visible or whether problems occurred during processing (e.g., access errors in locked directories).\nImplementing these measures consistently allows you to detect and evaluate security-relevant incidents during ongoing operations promptly. Furthermore, a transparent logging strategy ensures that administrators and incident response teams have a solid database to rely on during an investigation. This strengthens your application\u0026rsquo;s ability to respond to attacks and fulfils key requirements for traceability, auditing, and compliance in security-critical IT systems.\nSummary # We developed a functional file management application in Vaadin Flow in just a few steps. Users can upload files securely stored in the server directory and download them again. We used Java NIO to make file handling more efficient and secure. Additionally, we implemented safeguards against CWE-22 (Path Traversal), CWE-377 (Insecure Temporary File), and CWE-778 (Insufficient Logging) to ensure that attackers cannot gain unauthorised access to the file system, that temporary files are created securely, and that security-relevant events are comprehensively logged. This example demonstrates how easy it is to create a powerful user interface and implement server-side logic in Java with Vaadin Flow.\nThe next steps could be to extend the application, for example, by adding user permissions, customising the file overview, or integrating a search function for uploaded files. Vaadin offers numerous visual components to further improve the usability of your applications, but it is up to the developer to secure these features. Furthermore, we could extend the application with features such as drag-and-drop for file uploads, user role management, or a connection to a database for indexing files and storing metadata. However, with each additional feature, there are also additional attack vectors. It\u0026rsquo;s always important to find the right balance.\nBy enhancing security measures such as implementing robust file validation, sufficient logging mechanisms, and using Java NIO, we ensure that the application remains secure and efficient for both the developer and end users.\nWe\u0026rsquo;ll explore different aspects in the following parts, so it remains exciting.\nHappy Coding\nSven\n","date":"20 May 2025","externalUrl":null,"permalink":"/posts/creating-a-simple-file-upload-download-application-with-vaadin-flow/","section":"Posts","summary":"Vaadin Flow is a robust framework for building modern web applications in Java, where all UI logic is implemented on the server side. In this blog post, we’ll make a simple file management application step by step that allows users to upload files, save them to the server, and download them again when needed. This is a great way to demonstrate how to build protection against CWE-22, CWE-377, and CWE-778 step by step.\n","title":"Creating a simple file upload/download application with Vaadin Flow","type":"posts"},{"content":"","date":"11 April 2025","externalUrl":null,"permalink":"/tags/bytecode/","section":"Tags","summary":"","title":"Bytecode","type":"tags"},{"content":"","date":"11 April 2025","externalUrl":null,"permalink":"/tags/instrumentation-api/","section":"Tags","summary":"","title":"Instrumentation Api","type":"tags"},{"content":" What is the Java Instrumentation API? # The Java Instrumentation API is part of the java.lang.instrument package and allows you to change or analyse class bytecode at runtime. It is particularly intended for the development of profilers, agents, monitoring tools, or even dynamic security mechanisms that need to intervene deeply in a Java application\u0026rsquo;s behaviour without changing the source code itself.\nWhat is the Java Instrumentation API? What are the advantages and disadvantages? Advantages Disadvantages What are the performance implications? Influence during class loading Impact at runtime Memory and garbage collector effects What security implications arise? Security implications in detail Relevant CWEs (Common Weakness Enumerations) A practical example 1. REST service without external dependencies 2. Java Agent for instrumentation How does the whole thing start? Why do you need one? MANIFEST.MF? What does a valid one look like? MANIFEST.MF out of? Where and how is the file created? How does this work at runtime? What and why is this happening? At the heart of this API is the concept of Java Agents. An agent is a special component that is activated when the JVM starts or at runtime (via the so-called Attach API) can be loaded. The agent then uses an Instrumentation-Interface passed by the JVM launcher. This interface allows the agent to inspect already loaded classes, transform new classes or influence the behavior of the JVM - for example, by inserting hooks, measuring call times or injecting security checks.\nThe basic task is to transform the bytecode during the classloading process without violating the semantics of the Java language. Technically, this is done by a so-called ClassFileTransformer, which is registered and can manipulate the bytecode every time a class is loaded before it is interpreted or compiled by the JVM.\nA typical entry point for a static agent is a class with a premain(String agentArgs, Instrumentation inst)-Method. With dynamic agents, this comes instead agentmain(String agentArgs, Instrumentation inst)method to use. Both variants allow transformers to be registered and thus influence the behavior of the application below the language level.\nUsing the Instrumentation API requires a deep understanding of the JVM architecture and Java class loading mechanism. At the same time, it also opens up unique possibilities for runtime analysis and modification that would not be achievable using normal means in Java, and in a way that can be elegantly integrated into existing applications without having to recompile them.\nWhat are the advantages and disadvantages? # Advantages # A key advantage lies in the ability to manipulate bytecodes transparently : You can change the behavior of existing classes without touching their source code. This is particularly relevant in safety-critical or highly dynamic environments, such as when inserting logging, tracing or metrics code. The API also enables the so-called Class Retransformation , which means that already loaded classes can be subsequently changed under certain conditions. This feature is used intensively in conjunction with tools such as JRebel or modern observability solutions such as OpenTelemetry.\nAnother advantage is integration without source code changes. An agent can be started using the Java process or added at runtime without the target application even knowing it is being instrumented. This is particularly essential for debugging, monitoring or security solutions.\nIn addition, the Instrumentation API has been Part of the JDK since Java 5 and requires no external dependencies in security-conscious environments—for example, ​​critical infrastructure or the government environment—this can be a significant argument for but also against its use.\nDisadvantages # The power of the Instrumentation API inevitably brings significant disadvantages. First and foremost is the Complexity of bytecode manipulation. Although there are libraries such as ASM or Byte Buddy that make working with bytecode easier, getting started remains deep in the JVM\u0026rsquo;s engine room and requires a sound understanding of the Java class structure, class loading, and bytecode specification.\nA second risk concerns Stability and predictability : Interference with the bytecode can lead to subtle errors, for example, when method signatures are changed, security checks are bypassed, or synchronisation blocks are manipulated. Such errors are difficult to test and can only occur under certain runtime conditions, drastically increasing the debugging effort.\nThose too Compatibility across different JVM versions is a problem. Not every JVM behaves identically when it comes to class loading or transformation. Particularly with GraalVM or in AOT environments, certain features of the Instrumentation API may not work or only work to a limited extent. This particularly applies to dynamic reloading of classes or access to low-level JVM internals.\nAfter all, performance overhead should not be neglected. While simply registering an agent is usually very efficient, when transformers perform complex operations on every classload or inject metrics, this can result in measurable startup or runtime delays, especially in I/O-intensive server applications.\nThe Java Instrumentation API is a double-edged sword: It offers a unique opportunity to intervene in the behavior of a Java application deeply, but at the price of increased complexity, potential instability, and difficult maintainability. Its use should, therefore, be well justified, targeted, and accompanied by appropriate testing and logging infrastructure. However, it remains an indispensable tool in safety-critical or highly dynamic contexts where application code modification is impossible.\nWhat are the performance implications? # The performance implications of the Java Instrumentation API are complex and depend heavily on the respective use case, the complexity of the transformations carried out, and the type of target application. The Instrumentation API influences performance during three phases: during class loading, at runtime of the instrumented classes, and in connection with memory or security-relevant operations.\nInfluence during class loading # The most significant immediate impact occurs when the class loader loads a class. As soon as a ClassFileTransformer has been registered, it is called every time a new class is loaded. At this moment, the JVM passes the class\u0026rsquo;s original bytecode to the Transformer, which can modify it or return it unchanged. The duration of this transformation directly affects the application\u0026rsquo;s start time or the dynamic reloading of classes.\nComplex transformations, especially those performed with ASM or similar bytecode libraries, can quickly generate measurable overhead, especially in framework-based applications with thousands of classes. If additional logging, tracing or code injections are carried out in each method, the loading time increases exponentially because each method block has to be processed individually.\nImpact at runtime # Additional overhead may also arise at runtime, depending on how the bytecode was changed. This is particularly true if the Transformer inserts additional control flow, such as logging instructions, time measurement, additional validations or security-related checks. For example, such insertions can extend the execution of each method by several nanoseconds to milliseconds, which can lead to massive performance losses in methods called at high frequencies (e.g. in I/O loops or business logic).\nA typical example is the so-called method entry/exit tracing, where logging code is injected before and after each method. Even if this logging code itself does not actively write, but only sets passive markers, a memory and CPU load can still add up under high load.\nIn addition, the instrumentation can also JIT optimizations affect the JVM , especially when unpredictable control flows are inserted or methods are artificially inflated. This can lead to certain hotspots no longer being converted into natively optimised code, with corresponding consequences for execution time.\nMemory and garbage collector effects # Introducing additional object state during instrumentation—for example, through static caches, trace information, or associated metadata—can increase heap usage. This becomes particularly critical if weak or soft references are not managed correctly or if new classes with slightly different structures are continually created, which, in the worst case, can lead to an OutOfMemoryError in the class memory.\nThe Java Instrumentation API, when used efficiently and purposefully, can be used with minimal impact on performance. However, this requires that the registered transformers are kept as lean as possible, unnecessary transformations are avoided, and classes are filtered in a targeted manner. At the same time, one should always keep in mind that even small changes in the bytecode can have far-reaching effects on the JVM\u0026rsquo;s optimization strategies and memory behavior. Therefore, careful benchmarking and targeted monitoring are essential if you want to use the API in production systems.\nWhat security implications arise? # Because it can manipulate classes at runtime, remove security checks, or inject new bytecode, it opens up an attack surface far beyond what typical application code can do. It therefore represents a potential gateway for privilege escalation, code injection and persistence mechanisms, such as those observed with Advanced Persistent Threats (APT) or malicious agents.\nSecurity implications in detail # First of all, it should be noted that the Instrumentation API can deliberately operate past the JVM sandbox. Once an agent is active - be it when it is started via the -javaagent argument or dynamically via the Attach API - it has almost complete control over all loaded classes in the context of the running JVM. It can remove security checks, alter stack traces, redirect control flow, or extract sensitive information from memory. Also, the transformation of classes from security-relevant packages such as java.lang.reflect or javax.crypto is possible if no SecurityManager restrictions are active (which, however, have been deprecated since Java 17 and are completely eliminated from Java 21).\nThe Attach API is particularly critical because it allows arbitrary Java processes on the local system to be provided with an agent at runtime—as long as the same user context is present. This means that an attacker who gains access to a shell account could compromise any of that user\u0026rsquo;s Java processes without that process having to be prepared for this. This scenario corresponds to a form of runtime privilege escalation at the process level.\nRelevant CWEs (Common Weakness Enumerations) # Concerning the Java Instrumentation API, several CWE categories can be named that are relevant in the context of JVM and agent manipulation:\n**CWE-94: Improper Control of Generation of Code (\u0026lsquo;Code Injection\u0026rsquo;) **Suppose an agent is used to generate or load new bytecode from external sources at runtime (e.g., network data, configuration files, or user input). In that case, this is a classic code injection, which can lead to a full takeover of the JVM.\n**CWE-250: Execution with Unnecessary Privileges **By default, agents operate with the full rights of the Java process. Malicious agents can exploit unnecessary privileges if appropriate access protection is not implemented (e.g., through user separation or restrictive process isolation).\n**CWE-269: Improper Privilege Management **Improper use of an agent to transform security-critical classes (e.g. ClassLoader, AccessController, Cryptographic Provider) can lead to a breach of the Java security model.\n**CWE-272: Least Privilege Violation **If applications unnecessarily activate the Attach API or even provide their agent interfaces, this violates the principle of minimum rights allocation.\n**CWE-749: Exposed Dangerous Method or Function **An improperly configured application, for example, making the Attach API available via an HTTP endpoint (e.g. in self-built monitoring solutions) opens dangerous functions to external attackers.\n**CWE-798: Use of Hard-coded Credentials **In some dynamic agents, the transformation of security-relevant classes uses internal access or passwords that remain in the bytecode. These are easily extracted and can lead to the compromise of other systems.\nFrom a security perspective, the Java Instrumentation API can be classified as a privileged interface, which, if handled incorrectly, leads to significant opportunities for attacks. Your use should therefore always be carried out with procedural and technical controls flanked - for example, by isolating agents in dedicated containers, by restrictive file system and user rights, and by consistently deactivating the Attach API in production environments using the JVM option -Dcom.sun.management.jmxremote=false or -XX:+DisableAttachMechanism.\nCareful use of instrumentation is essential, particularly in security-critical scenarios, such as processing personal data or in distributed microservices architectures. Using this API without a sound security concept can quickly become a high-risk attack vector.\nA practical example # A simple but well-founded example of using the Java Instrumentation API in the context of a REST service can be implemented using onboard tools in the JDK, particularly using the com.sun.net.httpserver.HttpServer for server functionality. The goal is to write a Java agent that transforms all REST endpoints during class loading so that they automatically insert logging statements at the start of each method without changing the endpoints\u0026rsquo; original code.\nThe application,, therefore, consists of two parts:\nThe REST service based on HttpServer A Java agent that works via Instrumentation, a ClassFileTransformer registered 1. REST service without external dependencies # The aim of this service is to provide simple license management based on GET and POST requests:\nimport com.sun.net.httpserver.HttpServer; import com.sun.net.httpserver.HttpHandler; import com.sun.net.httpserver.HttpExchange; import java.io.IOException; import java.io.OutputStream; import java.net.InetSocketAddress; public class LicenseRestServer { public static void main(String[] args) throws IOException { HttpServer server = HttpServer.create(new InetSocketAddress(8080), 0); server.createContext(\u0026#34;/license\u0026#34;, new LicenseHandler()); server.setExecutor(null); server.start(); System.out.println(\u0026#34;Server started on port 8080\u0026#34;); } static class LicenseHandler implements HttpHandler { @Override public void handle(HttpExchange exchange) throws IOException { if (\u0026#34;GET\u0026#34;.equals(exchange.getRequestMethod())) { String response = \u0026#34;Current license: DEMO-1234-5678\u0026#34;; exchange.sendResponseHeaders(200, response.getBytes().length); try (OutputStream os = exchange.getResponseBody()) { os.write(response.getBytes()); } } else { exchange.sendResponseHeaders(405, -1); // Method Not Allowed } } } } This application starts a local REST server and responds GET /license with a static license key.\n2. Java Agent for instrumentation # Now, a static Java agent is implemented that runs when the JVM starts -javaagent can be integrated. The goal is to have the class at runtime LicenseHandler, to transform and to precede each method with simple logging.\nimport java.lang.instrument.ClassFileTransformer; import java.lang.instrument.Instrumentation; import java.security.ProtectionDomain; public class SimpleLoggingAgent { public static void premain(String agentArgs, Instrumentation inst) { System.out.println(\u0026#34;[Agent] Initializing agent with arguments: \u0026#34; + agentArgs); inst.addTransformer(new LoggingTransformer()); } static class LoggingTransformer implements ClassFileTransformer { @Override public byte[] transform(Module module, ClassLoader loader, String className, Class\u0026lt;?\u0026gt; classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) { if (className != null \u0026amp;\u0026amp; className.contains(\u0026#34;LicenseRestServer$LicenseHandler\u0026#34;)) { System.out.println(\u0026#34;[Agent] Transforming class: \u0026#34; + className); // Here you could modify real bytecode (e.g. with ASM) // For the purposes of this example, we return the original bytecode. } return classfileBuffer; } } } In this minimal form, the transformer does not make any real bytecode changes, but it shows the basic process: when loading the class LicenseHandler the transform method is called. An extended version could be done here using the ClassReader and ClassWriter out of org.objectweb.asm Inject targeted logging at the beginning of each method.\nHow does the whole thing start? # Compiling both classes into separate JARs: * license-service.jar contains the application. * agent.jar contains the agent and a MANIFEST.MF with Premain-Class: SimpleLoggingAgent. Starting the application with the agent: java -javaagent:agent.jar -jar license-service.jar\nWhen starting, the JVM loads the agent, then registers the transformer. As soon as the LicenseHandlerclass is loaded, the transformer gets access to its bytecode.\nThe Manifest file plays a central role in the context of a Java agent because it explicitly tells the JVM when it starts which class initializes the agent – i.e. where the method is premain() or agentmain() located. Without this information, the JVM cannot recognize the agent class and will ignore the agent at startup. This is the technical reason why the manifest file is mandatory if you want to build a Java agent.\nWhy do you need one? MANIFEST.MF? # If an agent over -javaagent:path/to/agent.jar is started, the JVM reads the JAR and searches in META-INF/MANIFEST.MF by one of the following attributes:\nPremain-Class : Specifies the class that will be used at JVM startup (before main()) is executed.\nAgent-Class : Specifies the class that belongs to. A more dynamic Agent attachment is used via the Attach API.\nCan-Redefine-Classes, Can-Retransform-Classes, etc. : Optional, to specify additional capabilities.\nWithout a correctly placed one Premain-Class attribute, the agent cannot be started. The JVM has no mechanism to guess or scan this automatically.\nWhat does a valid one look like? MANIFEST.MF out of? # A minimal manifest file for a static Java agent might look like this:\nManifest-Version: 1.0 Premain-Class: com.svenruppert.securecoding.agent.SimpleLoggingAgent Can-Redefine-Classes: true Ensure that each line has a line break (\\n) end and that no line break is forgotten. Spaces are also meaningful – there must be exactly one space between the attribute and the value.\nWhere and how is the file created? # You can create the MANIFEST.MF file in several ways. In this example, I\u0026rsquo;m using Maven to declare the necessary parameters and attributes.\nExample for pom.xml :\n\u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;artifactId\u0026gt;maven-jar-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;archive\u0026gt; \u0026lt;manifestEntries\u0026gt; \u0026lt;Premain-Class\u0026gt;com.svenruppert.securecoding.agent.SimpleLoggingAgent\u0026lt;/Premain-Class\u0026gt; \u0026lt;Can-Redefine-Classes\u0026gt;true\u0026lt;/Can-Redefine-Classes\u0026gt; \u0026lt;/manifestEntries\u0026gt; \u0026lt;/archive\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; How does this work at runtime? # If you start the JVM with the agent:\njava -javaagent:simple-agent.jar -jar my-rest-service.jar\nThe JVM does the following:\nRead the JAR simple-agent.jar. Opens META-INF/MANIFEST.MF. Finds Premain-Class: SimpleLoggingAgent. Loads this class. Leads premain(String args, Instrumentation inst) out, still before the main()method of the actual program is started. This gives the agent the chance to influence the behavior of the program before the first line of business logic is executed. This is precisely where his power lies – and the risk. The manifest file is the link between the JVM startup mechanism and the agent code. It is essential for Java agents because it tells the JVM which code actually contains agent logic. Their correct creation is a prerequisite for any form of static instrumentation. If you set it incorrectly or not at all, it will have no effect when the agent is started - the JVM will then ignore the agent without comment.\nWhat and why is this happening? # This architecture shows the strength of the Instrumentation API: without changing a line of application code, functionalities such as logging, tracing, or metrics can be injected at runtime. This happens early in the JVM\u0026rsquo;s life cycle (before the actual program starts), which means that all future classes can be affected if specifically addressed.\nEven without additional libraries, this setup demonstrates the core mechanism of the Instrumentation API: runtime manipulation of classes in a production-level environment. It shows how profound and dangerous this mechanism can be if there are no appropriate protection mechanisms at the JVM or operating system level.\nHappy Coding\nSven\n","date":"11 April 2025","externalUrl":null,"permalink":"/posts/open-hearted-bytecode-java-instrumentation-api/","section":"Posts","summary":"What is the Java Instrumentation API? # The Java Instrumentation API is part of the java.lang.instrument package and allows you to change or analyse class bytecode at runtime. It is particularly intended for the development of profilers, agents, monitoring tools, or even dynamic security mechanisms that need to intervene deeply in a Java application’s behaviour without changing the source code itself.\n","title":"Open-hearted bytecode: Java Instrumentation API","type":"posts"},{"content":"","date":"8 April 2025","externalUrl":null,"permalink":"/tags/parallel-programming/","section":"Tags","summary":"","title":"Parallel Programming","type":"tags"},{"content":"","date":"8 April 2025","externalUrl":null,"permalink":"/categories/streams/","section":"Categories","summary":"","title":"Streams","type":"categories"},{"content":"","date":"8 April 2025","externalUrl":null,"permalink":"/tags/streams/","section":"Tags","summary":"","title":"Streams","type":"tags"},{"content":"Sometimes it\u0026rsquo;s not enough for something to work - it has to work under load. In modern applications that process large amounts of data, the Streams API in Java provides developers with an elegant, declarative tool to transform, filter, and aggregate data in pipelines. ​​Describing complex data operations with just a few lines is seductive and realistic. But what happens when these operations encounter millions of entries? When should execution be done in multiple threads in parallel to save time and effectively use multi-core systems?\nBasics: Collector and concurrency Criteria for parallelizable collectors Examples from the Java standard library Own parallel collector implementations Best practices for productive use Application scenario: License analysis in the benchmark The data model The test setup Comparison with alternative parallelism concepts Conclusion and outlook At this point, a concept that often receives too little attention is the collector. The element at the end of a stream pipeline determines what should happen with the processed data. And even though the API seems simple – collect(Collectors.toList()) – there is an architecture behind it that brings its challenges when executed in parallel.\nThis text is, therefore, not just about the syntax or mechanics of collectors but rather a deep understanding of the conditions under which they can be used correctly and efficiently in parallel. We look at standard JDK solutions, discuss our implementations, show typical errors, and ultimately ask ourselves: How parallel can a collector be without becoming dangerous?\nBasics: Collector and concurrency # At first glance, Java\u0026rsquo;s Streams API suggests that collecting results—the so-called terminal aggregation—can easily be carried out in parallel. But behind the method collect(\u0026hellip;) hides more than just syntactic convenience. It is a coordinated collaboration between a data stream and a so-called collector—an object that forms a whole from individual parts.\nA collector essentially consists of four functional components: the supplier, which provides a new cache for each subprocess; the accumulator, which feeds elements into this cache; the combiner, which merges multiple caches into one; and finally, the finisher, which produces the result. While supplier and accumulator are also essential in sequential streams, the combiner only comes into action when several threads have collected independently of each other, i.e., one parallelStream().\nHere lies the first fundamental difference between sequential and parallel processing: in a sequential stream, it is sufficient to accumulate step by step into a single memory. In the parallel variant, on the other hand, several buffers are created that are isolated from one another, the contents of which must later be merged into a final result without conflict. This merging happens through the combiner, and it is precisely at this point that it is decided whether a collector is suitable for parallel use or not.\nThis suitability depends on several properties: The operations must be associative, i.e., deliver the same result regardless of the combination of the intermediate results. In addition, there must not be a shared state without synchronization. Last but not least, the individual steps must remain deterministic and free of side effects—otherwise, parallelisation quickly becomes a source of subtle errors.\nKnowing these structural requirements is the first step toward consciously using parallel processing. Only if you understand how the collector and stream work together can you estimate when a performance gain is possible and when you will get unstable or simply wrong results instead.\nCriteria for parallelizable collectors # Let\u0026rsquo;s imagine a stream running in parallel—perhaps over a large dataset divided into several segments. Each segment is now processed independently. What sounds trivial has profound implications: As soon as several threads collect at the same time, their intermediate results must not get in the way. The responsibility for correctness lies with the collector—or, more precisely, with its structural and functional design.\nThe first fundamental property is Associativity. A combiner call must produce consistent results regardless of the order. combine (a, b) and combine (b, a) must produce equivalent results. This is necessary because the order of combination in a parallel context depends on the scheduler and is, therefore, unpredictable.\nA second central point concerns Access to memory structures. A potential hotspot for race conditions arises when a collector uses a shared, mutable state during accumulation, such as an out-of-sync list or map. The collector must either work exclusively with local, thread-isolated caches or rely on concurrent data structures such as b: ConcurrentHashMap, LongAdder, or explicitly synchronized wrappers.\nIn addition, it is also deterministic and an essential criterion: parallel execution must not lead to different results, neither in terms of content nor structure. Caution is advised, especially with non-deterministic structures like HashSet or HashMap, as the iteration order can vary, as with Collectors.joining() or Collectors.toMap() becomes problematic when the application relies on order.\nThese requirements—associativity, isolated state, and determinism—form the technical touchstone for parallel collectors. They are not optional but fundamental. Anyone who ignores them risks difficult-to-reproduce errors, incomplete results, or high-performance but semantically incorrect output.\nConcurrency in Java Streams is powerful but not trivial. Only if a collector meets these criteria does it become one parallelStream() more than a placebo effect.\nExamples from the Java standard library # An obvious way to make the abstract concept of parallel collectors tangible is through the collectors already included in the Java standard library. Many developers use Collectors.toList(), toSet(), or joining() almost every day—but rarely with knowledge of whether and how these collectors behave in a parallel context.\nA simple example: The Collector Collectors.toList() uses an internal ArrayList. This is not thread-safe; therefore, the result when used in parallel is potentially inconsistent unless the buffers are isolated internally.\npublic static \u0026lt;T\u0026gt; Collector\u0026lt;T, ?, List\u0026lt;T\u0026gt;\u0026gt; toList() { return new CollectorImpl\u0026lt;\u0026gt;(ArrayList::new, List::add, (left, right) -\u0026gt; { left.addAll(right); return left; }, CH_ID); } In fact, this collector still works correctly in parallel streams because the Streams API allocates each thread its accumulation area and merges only at the end via a combined merge process. The crucial point lies not in the data structure itself but in its controlled isolation.\nIt turns out to be less robust: Collectors.groupingBy(\u0026hellip;). This variant is based on one HashMap and is not designed for concurrent access. This collector will be in one parallelStream(). Used without suitable protective measures, there is a risk of race conditions. The standard solution is Collectors.groupingByConcurrent(\u0026hellip;), internal ConcurrentHashMap, designed for simultaneous access.\npublic static \u0026lt;T, K\u0026gt; Collector\u0026lt;T, ?, ConcurrentMap\u0026lt;K, List\u0026lt;T\u0026gt;\u0026gt;\u0026gt; groupingByConcurrent(Function\u0026lt;? super T, ? extends K\u0026gt; classifier) { return groupingByConcurrent(classifier, ConcurrentHashMap::new, toList()); } A look at the signature of this method already shows the intention:\nMap\u0026lt;Integer, List\u0026lt;String\u0026gt;\u0026gt; result = namen.parallelStream() .collect(Collectors.groupingByConcurrent(String::length)); In this example, strings are grouped by their length in a way that can be processed in parallel. Crucially, both the map implementation and the accumulation process are thread-safe.\nEqually interesting Collectors.toConcurrentMap(\u0026hellip;), which is explicitly intended to aggregate large sets of key-value pairs in parallel. The combination of key conflicts and the correct handling of merge functions is particularly interesting here.\nThese examples conclude that not every standard collector is suitable for parallel use per se. Just because a method from the Collectors-kit, this does not mean that it will work correctly in every execution configuration. The context decides - and with it the data structure used, the behavior of the combiner and the type of accumulation.\nSo, if you want to get more than one result from a stream—a correct and high-performance result—you should choose your collector just as carefully as you choose the filter criterion at the beginning of the pipeline.\nOwn parallel collector implementations # As powerful as the Java Standard Library\u0026rsquo;s prebuilt collectors are, they are sometimes not sufficient for specific needs. Especially when domain-specific aggregations, specialized data structures, or non-trivial reduction logic are required, it is worth considering the possibility of creating your own collector implementations.\nA custom collector is usually created using the static method Collector.of(\u0026hellip;) created. This method expects five parameters: one Supplier, which creates a new accumulator; a BiConsumer\u0026lt;A, T\u0026gt;, which inserts an element into the accumulator; a BinaryOperator for combining two accumulators; optionally one Function\u0026lt;A, R\u0026gt; to convert the result; and a Collector.Characteristics\u0026hellip;-Array containing meta information like CONCURRENT or UNORDERED provides.\nA simple but meaningful collector could, for example, collect strings in parallel ConcurrentLinkedQueue collect:\nCollector\u0026lt;String, ?, Queue\u0026lt;String\u0026gt;\u0026gt; toConcurrentQueue() { return Collector.of( ConcurrentLinkedQueue::new, Queue::add, (left, right) -\u0026gt; { left.addAll(right); return left; }, Collector.Characteristics.CONCURRENT, Collector.Characteristics.UNORDERED ); } This collector is both CONCURRENT and UNORDERED, which means it can be written to by multiple threads simultaneously without guaranteeing the insertion order. What is important is that ConcurrentLinkedQueue acts as a thread-safe data structure, and the operation addAll is also incidentally uncritical.\nHowever, more complex scenarios, such as the parallel determination of statistical key figures (minimum, maximum, average) across a data set, are also conceivable. In such cases, one can record serve as an accumulator structure that already encapsulates all the required partial states. The combiner then only has to consolidate these structures field by field.\nOwn collector implementations force you to deal intensively with the parallelizability of the data structures used and the combinability of the aggregation logic. This is not a disadvantage, but a valuable learning effect, because only those who understand what a collector does inside can use it consciously and safely.\nBest practices for productive use # Anyone who wants to use collectors productively in a parallel context should consider some proven strategies - not as rigid rules, but as a framework for robust and efficient implementations.\nA first principle is: Only parallelize if there is real benefit to be expected. Small amounts of data, trivial transformations or IO-bound processes usually do not benefit from parallelStream(). On the contrary: the overhead of thread management can even exceed the potential performance gain. Parallelization is only worthwhile if the amount of data to be processed is sufficiently large and the operations are CPU-intensive.\nSecond: Use only thread-safe or isolated data structures. This means that each thread uses its accumulator, which the Streams API supports internally, or explicitly concurrent data structures such as ConcurrentHashMap, ConcurrentLinkedQueue, or atomic wrappers can be used.\nThird: Select collectors specifically. The standard library provides groupingByConcurrent, toConcurrentMap or mapping powerful tools specifically designed for parallel use. Anyone who also develops their solutions should pay particular attention to this combiner and the associativity of logic.\nFourth: Validate results , especially for new or complex pipelines. Parallel streams do not behave deterministically in execution, so tests in different utilization scenarios and under varying loads are necessary. This is especially true if collectors were developed or adapted in-house.\nLast but not least: Measure instead of assume. Tools such as JMH (Java Microbenchmark Harness), Flight Recorder, or async-profiler help make realistic statements about performance advantages. Parallelization without metrics is like flying blind with a tailwind—it may be faster, but possibly in the wrong direction.\nIn summary, Parallel collectors are powerful tools, but they can only develop their potential if they are used consciously, well-founded, and with an eye on data structure, semantics, and execution context.\nApplication scenario: License analysis in the benchmark # In our example, we will now look at the following scenario. We will evaluate fictitious license information. These consist of several attributes that lend themselves to automated analysis: Is the license valid? Is this a trial version? Which country does the license come from? How long was it valid?\nInstead of answering these questions individually and sequentially, the Java Streams API allows us to view the data as a stream and convert it into a consolidated analysis in a collector. But what happens if we analyse not 100, but 100,000 or 1,000,000 such license objects? A simple one is enough stream(), or is the parallel variant worth it? parallelStream()?\nThis is precisely where our experiment comes in: We want to measure the amount of data for which a parallel collector is noticeably worthwhile, both in terms of runtime and resource utilization. We use a specially developed collector that aggregates various metrics:\nTotal number of valid licenses Share of test licenses Average duration in days Distribution of licenses by country The oldest and youngest licenses issued To capture these values ​​as realistically as possible, we use a benchmark with JMH, the official microbenchmark tool of the OpenJDK community. The aim is to develop a sense of when parallel collectors actually offer added value and under which conditions they may even prove inefficient.\nThe data model # The benchmark\u0026rsquo;s data model is based on a practical license management system, such as those typically used in software products, platform services, or license servers. It consists of a compact but expressive Record class called LicenseInfo, which encapsulates all the essential information of a single software license.\nAt the center is the precise product allocation across the field productId, which allows conclusions about the licensed application or component to be drawn. The license itself is replaced by a cryptographically unique licenseKey represented, simulated here in the example, by a randomly generated UUID. These two fields are responsible for the formal identification of each license instance.\nTwo further fields describe temporal aspects: issuedAt indicates the date and time of issuance, and during validUntil marks the expiry time. This not only allows you to check whether a license is currently valid, but also the period for which it was originally created. Based on this period, the average useful life can be calculated, a central metric type for license analyses in companies.\nThe structure is complemented by the field country, which documents the geographical origin or scope of the license. This is relevant for compliance issues, regional licensing models or sales evaluations. The information is intended as an ISO country abbreviation, but can be flexibly adapted to actual requirements.\nFinally, the field isTrial depends on whether it is one Trial license or a regular license. This boolean field enables differentiated evaluations according to usage type—an important aspect if, for example, a distinction is to be made between active customers and evaluations.\nThe record definition remains deliberately slim, but provides a solid basis for various analysis methods: from counting and grouping to time series and country statistics. Their immutability and precise semantics make them ideal for use in parallel streams and collecting aggregations. In combination with a user-defined collector, the model allows precise and at the same time high-performance evaluation of extensive license stocks, with full parallelizability.\nimport java.time.LocalDateTime; public record LicenseInfo( String productId, String licenseKey, LocalDateTime issuedAt, LocalDateTime validUntil, String country, boolean isTrial ) {} The test setup # The structure of the benchmark for license analysis follows a systematic pattern, which is typical for microbenchmark-based performance tests with JMH. The aim is to measure under controlled conditions how the running time of a specially developed collector changes as the amount of data increases, both in serial and in parallel execution. The focus is not on comparing different algorithms, but rather on the stream processing strategy\u0026rsquo;s effect on the aggregation\u0026rsquo;s overall duration.\nThe test focuses on the benchmark class LicenseCollectorBenchmark, which is equipped with the annotations of the JMH library. It defines two measuring points: serialCollector() for serial evaluation using stream() and parallelCollector() for parallel evaluation using parallelStream(). Both methods process identical data but use different execution strategies.\nBefore each benchmark run, the @Setup(Level.Iteration)-annotated method setUp() generates a consistent test data set. This amount of data – 100,000 randomly generated by default LicenseInfo-Instances - is built fresh with each iteration to eliminate possible caching effects or side effects between benchmarks. The values ​​such as product ID, license key, timestamp, country code and test status are generated randomly, creating realistic heterogeneity.\nBoth benchmark methods then fully aggregate the license data, each using the same custom collector. This collector consolidates the data into one final LicenseStats-Object that summarizes several metrics, including the number of valid licenses, the number of test licenses, the average term, the country-specific distribution and the earliest and latest issuance time. This breadth of aggregation ensures that the collector in the benchmark is sufficiently loaded both computationally and in memory terms.\nThe actual measurement is carried out via the JMH runtime mechanism with a predefined number of warm-up and measurement cycles. The average execution time per benchmark method is expressed in milliseconds per operation (ms/op), outputting an easily interpretable value representing the total stream processing duration, including accumulation and reduction.\nThis setup makes it possible to quantify the efficiency advantages – or potential disadvantages – of parallel stream processing under practical conditions. The comparability between serial and parallel is complete because the data is identical, the collector is similar, and only the execution strategy differs.\nimport org.openjdk.jmh.annotations.*; import java.time.LocalDateTime; import java.time.Duration; import java.util.*; import java.util.concurrent.*; import java.util.stream.Collector; import java.util.stream.Collectors; import static java.util.concurrent.TimeUnit.MILLISECONDS; @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(MILLISECONDS) @State(Scope.Thread) public class LicenseCollectorBenchmark { static final int DATA_SIZE = 100_000; List\u0026lt;LicenseInfo\u0026gt; licenseData; @Setup(Level.Iteration) public void setUp() { Random rand = new Random(); licenseData = new ArrayList\u0026lt;\u0026gt;(DATA_SIZE); for (int i = 0; i \u0026lt; DATA_SIZE; i++) { licenseData.add(new LicenseInfo( \u0026#34;product-\u0026#34; + rand.nextInt(100), UUID.randomUUID().toString(), LocalDateTime.now().minusDays(rand.nextInt(1000)), LocalDateTime.now().plusDays(rand.nextInt(1000)), \u0026#34;country-\u0026#34; + rand.nextInt(20), rand.nextBoolean() )); } } @Benchmark public LicenseStats serialCollector() { return licenseData.stream() .collect(LicenseCollectors.stats()); } @Benchmark public LicenseStats parallelCollector() { return licenseData.parallelStream() .collect(LicenseCollectors.stats()); } // --- data model --- public record LicenseInfo( String productId, String licenseKey, LocalDateTime issuedAt, LocalDateTime validUntil, String country, boolean isTrial ) {} public record LicenseStats( long totalValid, long totalTrial, double avgDurationDays, Map\u0026lt;String, Long\u0026gt; byCountry, LocalDateTime oldestIssued, LocalDateTime newestIssued ) {} // --- Collector-Factory --- static class LicenseCollectors { static Collector\u0026lt;LicenseInfo, ?, LicenseStats\u0026gt; stats() { class Acc { long valid = 0; long trial = 0; long total = 0; long sumDays = 0; Map\u0026lt;String, Long\u0026gt; byCountry = new ConcurrentHashMap\u0026lt;\u0026gt;(); LocalDateTime oldest = null; LocalDateTime newest = null; void add(LicenseInfo li) { if (li.validUntil().isAfter(LocalDateTime.now())) valid++; if (li.isTrial()) trial++; long days = Duration.between(li.issuedAt(), li.validUntil()).toDays(); sumDays += days; total++; byCountry.merge(li.country(), 1L, Long::sum); if (oldest == null || li.issuedAt().isBefore(oldest)) oldest = li.issuedAt(); if (newest == null || li.issuedAt().isAfter(newest)) newest = li.issuedAt(); } Acc merge(Acc other) { valid += other.valid; trial += other.trial; sumDays += other.sumDays; total += other.total; other.byCountry.forEach((k, v) -\u0026gt; byCountry.merge(k, v, Long::sum)); if (oldest == null || (other.oldest != null \u0026amp;\u0026amp; other.oldest.isBefore(oldest))) oldest = other.oldest; if (newest == null || (other.newest != null \u0026amp;\u0026amp; other.newest.isAfter(newest))) newest = other.newest; return this; } LicenseStats finish() { double avg = total == 0 ? 0 : (double) sumDays / total; return new LicenseStats(valid, trial, avg, byCountry, oldest, newest); } } return Collector.of( Acc::new, Acc::add, Acc::merge, Acc::finish, Collector.Characteristics.UNORDERED, Collector.Characteristics.CONCURRENT ); } } } Comparison with alternative parallelism concepts # While parallel streams with collectors offer a declarative and comparatively accessible way to use multi-core processors, there are other parallelisation models in the Java platform, each with its own strengths and areas of application.\nAn obvious comparison is with the ForkJoinPool-API. This allows recursive tasks to be broken down and carried out efficiently in parallel, for example, when searching or sorting large amounts of data. However, unlike streams, fork/join structures are explicit: the developer manually controls the division of tasks and the combination of results. This creates more flexibility, but also more complexity and potential for errors.\nAnother approach is to use CompletableFuture – especially in combination with methods like supplyAsync() or thenCombineAsync(). The focus here is not on aggregating data but coordinating asynchronous, non-blocking tasks. While streams must have all the available data, they are suitable CompletableFutures for I/O-heavy or delayed processing.\nNewer Java versions also provide structures such as StructuredTaskScope, which comes into play - a still experimental but promising approach to organise parallel tasks better and explicitly control their lifespan. Unlike streams, this API allows structured concurrency with clearly defined error handling and cancellation semantics.\nParallel collectors position themselves within this spectrum as a compact, declarative mechanism focusing on data-centric, memory-bound processing. They offer low barriers to entry, but are limited in terms of expressiveness and controllability. They reach their natural limits in more complex scenarios - with dependencies between subtasks, adaptive scheduling or concurrent I/O operations.\nTherefore, the choice of the appropriate parallelism model should always be based on the task\u0026rsquo;s requirements: Where pure data aggregation is involved, streams and parallel collectors are ideal. However, where control flow, resilience, or the ability to integrate into distributed systems is required, other APIs are more effective.\nConclusion and outlook # Parallel collectors are not a sideshow to the Java Streams API - they are at the heart of whether modern computing can truly scale. Anyone who understands their structure quickly realizes that the correct aggregation in a parallel context is not a coincidence, but the result of precise planning. Associativity, deterministic processing and thread-safe data structures are not academic footnotes, but central pillars of this architecture.\nThe concepts shown here—from the structural basis to best practices to your own implementations—are intended not only to convey the functionality but also to create awareness of the responsibility that comes with parallelism. What is harmless in a single-threaded scenario can lead to serious errors in a parallel environment.\nThe question for the future is how the Streams API will continue to develop. More structured parallelism models, which are based on concepts such as structured concurrency or data flow graphs, are conceivable. Integration with reactive streams or declarative parallelism à la Fork/Join-DSL would also be a logical next step.\nOne thing is certain: Anyone who learns to use parallel collectors correctly and efficiently today will lay the foundation for high-performance and scalable data processing in the Java platform of tomorrow.\nHappy Coding\nSven\n","date":"8 April 2025","externalUrl":null,"permalink":"/posts/synchronous-in-chaos-how-parallel-collectors-bring-order-to-java-streams/","section":"Posts","summary":"Sometimes it’s not enough for something to work - it has to work under load. In modern applications that process large amounts of data, the Streams API in Java provides developers with an elegant, declarative tool to transform, filter, and aggregate data in pipelines. ​​Describing complex data operations with just a few lines is seductive and realistic. But what happens when these operations encounter millions of entries? When should execution be done in multiple threads in parallel to save time and effectively use multi-core systems?\n","title":"Synchronous in Chaos: How Parallel Collectors Bring Order to Java Streams","type":"posts"},{"content":"","date":"7 April 2025","externalUrl":null,"permalink":"/tags/dns/","section":"Tags","summary":"","title":"DNS","type":"tags"},{"content":"","date":"7 April 2025","externalUrl":null,"permalink":"/tags/dns---hijacking/","section":"Tags","summary":"","title":"DNS - Hijacking","type":"tags"},{"content":"","date":"7 April 2025","externalUrl":null,"permalink":"/tags/dns-attacks/","section":"Tags","summary":"","title":"DNS Attacks","type":"tags"},{"content":" 1. Getting started – trust in everyday internet life # Anyone who enters a web address like “www.example.de” into the browser expects a familiar website to appear within seconds. Whether in the home office, at the university, or in the data center, access to online services is now a given. The underlying technical processes are invisible to most users; even in IT practice, they are often taken for granted. One of these invisible processes is name resolution by the Domain Name System (DNS).\n1. Getting started – trust in everyday internet life 2. The Domain Name System – backbone of the digital world 3. Invisible Attack Surface – Why DNS is vulnerable 4. DNS cache poisoning – The classic attack 5. Amplification and Deception – DNS as a vehicle for DDoS and tunneling 6. DNS Hijacking – When attackers dictate the path 7. Attack detected – but how? 8. Protective measures – think and implement DNS securely 8.1 General Network Protection Measures 8.3 Advanced protection measures at protocol and application level 9. Conclusion – trust needs protection DNS is the phone book of the Internet. It ensures that human-readable domain names are translated into numeric IP addresses so that computers worldwide can communicate with each other. Without DNS, there would be no URLs, emails, or APIs. Nevertheless, DNS has had a shadowy existence in the security discourse for a long time. It is not a purely technical detail but rather a central link in the trust relationship between the user, application, and infrastructure.\nBecause DNS works in the background, it is a popular target for attackers. DNS attacks aim to disrupt, manipulate, or redirect this process for your own purposes, with potentially serious consequences. Anyone who relies on a domain name always leading to the correct IP address risks falling into a well-disguised trap. The affected applications appear to continue to work perfectly—they just communicate with the wrong counterpart.\nThis text highlights the diverse forms of attacks on DNS, how they work, and their implications for the operation and development of secure systems. Particular attention is paid to the question of how DNS must be considered from the perspective of modern applications, especially in Java, so as not to become the Achilles heel of an otherwise solid security architecture.\n2. The Domain Name System – backbone of the digital world # The Domain Name System has become integral to today\u0026rsquo;s Internet communication. It was developed in the early 1980s to replace the increasingly confusing web of growing hostnames with a scalable, hierarchical structure. Instead of remembering IP addresses, people can now use spoken domain names, and rely on them reliably pointing to the correct servers.\nThe basic job of DNS is to translate a string like “www.uni-heidelberg.de“ to an IP address like “129.206.100.168”. This process occurs in several steps and is handled by different types of servers. The process usually begins with the so-called resolver – a system service or external service provider that accepts requests from end devices. This resolver in turn queries root name servers, top-level domain servers (e.g. for .de, .com, .org) and finally authoritative name servers until it finds the IP address it is looking for.\nThe concept of caching is essential to DNS\u0026rsquo;s efficiency. Each resolver stores already resolved queries for a defined period of time to speed up re-queries. Operating systems and applications can also maintain their own DNS caches. This behavior contributes significantly to reducing global DNS traffic, but it is also one of the biggest weak points, as later chapters will show.\nDNS is based on the User Datagram Protocol (UDP) and is stateless and unencrypted by design. Although extensions such as DNSSEC or DNS-over-HTTPS exist, they are far from being used widely in everyday life. This means that DNS remains an open system, efficient but vulnerable. The assumption that DNS “just works” quickly becomes a dangerous illusion from a security perspective.\n3. Invisible Attack Surface – Why DNS is vulnerable # The architecture of the Domain Name System follows the paradigm of openness. This means that requests are standardised, unencrypted, and mostly stateless, which is precisely why they are particularly high-performance. But the very properties that make DNS a robust part of the Internet infrastructure also pose serious risks. The vulnerability of the DNS is not an accidental by-product of modern forms of attack but is rooted in the original system design.\nDNS was designed when the Internet was a trusted space characterised by academic collaboration and open standards. Security mechanisms were not part of the design goal at the time. Accordingly, DNS still lacks built-in integrity, authenticity, and confidentiality. Queries are not verified, answers are not signed, and the source is not checked. This creates attack vectors that are still being exploited today by actors of all stripes.\nAdditionally, DNS requests are typically transmitted over UDP, which makes them easy to forge or intercept. Transaction IDs and ports offer minimal protection against spoofing, but these hurdles are technically easy to overcome with today\u0026rsquo;s means. Even with TCP-based fallbacks, absolute protection is lacking without additional measures such as DNSSEC.\nA particular vulnerability lies in the caching behavior of resolvers and applications. Once saved, answers are considered valid for a certain period, regardless of whether they were correct or manipulated. Attackers can specifically exploit this behavior to inject false answers into the cache and thus redirect many subsequent requests. This so-called cache poisoning is one of the oldest but still most effective attacks on the DNS.\nDNS is therefore not a secure anchor of trust, at least not without additional measures. Anyone using DNS in a security-critical architecture, such as modern Java applications, cloud APIs or container infrastructures, should know the inherent weaknesses. DNS is robust, but it can also be manipulated. This makes it an attractive target – and an often underestimated attack surface.\n4. DNS cache poisoning – The classic attack # Among the numerous attack methods on the domain name system, DNS cache poisoning—also known as DNS spoofing—is considered particularly perfidious. The aim of this attack is to manipulate the caching of DNS responses. The attacker injects a fake DNS response into a resolver\u0026rsquo;s cache, with the effect that all subsequent queries to a specific domain name point to a false IP address. The affected users usually do not notice the attack: the website looks as usual, except that the attacker controls it.\nThe process of such an attack is as technically interesting as it is security-relevant. The attacker does not wait for a resolver to resolve a specific domain but proactively tries to outsmart the resolver with fake answers. To do this, it must guess the transaction ID of the DNS request - a short, 16-bit random number - and respond faster than the legitimate DNS server. If this succeeds, the resolver accepts the incorrect answer as valid and stores it in the cache.\nWhat is particularly critical is that this manipulation persists: As long as the cache entry remains valid (TTL), every new request will return the same fake answer. This way, individual users and entire organisations can be systematically misled - for example, on phishing sites, fake authentication services or malware sources. This can result in a REST API being accessed supposedly successfully in Java applications, even though a completely different infrastructure is responding in the background.\nModern DNS implementations now rely on additional protection mechanisms such as random source ports, rate limiting, or DNSSEC. But many resolvers—especially on local networks or outdated systems—remain vulnerable. Even browsers or libraries with built-in DNS caching can become targets.\nFor developers, this means that the trust assumption that InetAddress.getByName() always returns the “correct” address is dangerous. Explicit protective measures are required when DNS is used in security-critical contexts, for example, through the use of cryptographically signed DNS responses (DNSSEC), deliberate cache invalidation, or explicit validation of the remote site, for example, via TLS certificates.\nDNS cache poisoning is a lesson in how a helpful performance optimization—caching—can turn into the exact opposite. Anyone who wants to operate or use DNS securely must understand how the system works and how easily it can be manipulated at its core.\n5. Amplification and Deception – DNS as a vehicle for DDoS and tunneling # In addition to the targeted manipulation of DNS caches, there are other attack scenarios in which DNS serves as a technical vehicle for higher-level goals, such as denial-of-service attacks or secret data transmission. Two particularly relevant variants in this context are DNS amplification and DNS tunneling.\nA DNS amplification attack is a form of reflected denial of service (DoS) that exploits the asymmetric structure of the DNS protocol. The attacker sends a small DNS query - often of type ANY - to an open resolver, but spoofs the source address to point to the victim. The resolver responds with a significantly larger amount of data than the request, which is sent directly to the victim. Multiplied across many resolvers, this creates a massive data stream that is explicitly used to overload servers. DNS amplification is so effective because the attacker can cause disproportionate damage with minimal use of resources.\nAnother gateway is DNS tunneling. This technique uses DNS packets to pass any other data through firewalls or security measures. Since DNS traffic is hardly restricted in many networks, the protocol is ideal for bypassing control authorities. The payload—such as commands to malware or stolen data—is hidden in seemingly legitimate DNS queries or responses. On the receiving end, this information is extracted and evaluated again. DNS tunneling is considered particularly insidious because it is difficult to detect using traditional security tools.\nA simple Java example demonstrating how a DNS tunnel could be hidden in a seemingly legitimate HTTP communication can be implemented using Java SE\u0026rsquo;s on-board tools. In the following scenario, we implement a minimalist REST server with com.sun.net.httpserver.HttpServer, which receives DNS-like encoded data, representative of a possible C2 channel (Command \u0026amp; Control).\nHttpServer server = HttpServer.create(new InetSocketAddress(8080), 0);\nserver.createContext(\u0026quot;/api/lookup\u0026quot;, exchange - \u0026gt; {\n** if (\u0026ldquo;GET\u0026rdquo;.equals(exchange.getRequestMethod())) {**\n** URI requestURI = exchange.getRequestURI();**\n** String query = requestURI.getQuery();**\n** if (query != null \u0026amp;\u0026amp; query.contains(\u0026ldquo;q=\u0026rdquo;)) {**\n** String domainPayload = query.split(\u0026ldquo;q=\u0026rdquo;)[1];**\n** String decoded = domainPayload.replace(\u0026quot;.\u0026quot;, \u0026quot; \u0026ldquo;); // simplified payload decoding**\n** System.out.println(\u0026quot;[DEBUG] Empfangenes DNS-Tunnel-Payload: \u0026quot; + decoded);**\n** }**\n** String response = \u0026ldquo;OK\u0026rdquo;;**\n** exchange.sendResponseHeaders(200, response.length());**\n** exchange.getResponseBody().write(response.getBytes());**\n** exchange.close();**\n** }**\n});\nserver.setExecutor(null);\nserver.start();\nIn this simplified simulation, a DNS-like query parameter is received via a REST interface - e.g. b. http://localhost:8080/api/lookup?q=cmd.run.download. The server “interprets” the dot as a separator of an encoded DNS structure, as is common in DNS tunneling. In a real attack variant, this mechanism would be used by malware or scripts for bidirectional data exchange. What is critical here is that the transport takes place over seemingly unsuspicious channels - HTTP, DNS, ICMP - while the actual payload is hidden deep in protocol structures.\nBoth types of attacks have in common that DNS is not a direct target, but rather a tool - a transport system for superimposed attack targets. This results in the need for administrators and developers to monitor DNS traffic functionally and from a security perspective. In Java-based architectures, DNS dependencies should be consciously modeled, logged and – where possible – checked for validation and integrity. A simple forwarding of data via DNS may seem harmless, but it may be the start of a widespread attack.\n6. DNS Hijacking – When attackers dictate the path # While DNS cache poisoning focuses on manipulating cached responses, DNS hijacking targets the source itself: the DNS resolver or the infrastructure that controls it. The attack begins when users resolve their domain names - either on the local device, on the home router, on corporate networks, or via an ISP\u0026rsquo;s resolver.\nWith DNS hijacking, DNS requests are systematically redirected - for example, through manipulation of network settings, compromised firmware, malicious DHCP servers or malware that specifically changes the configuration of a system. The resolver then no longer returns the IP addresses expected by the user, but instead forwards requests to servers controlled by the attacker. In practice, this means: Even if a user “www.bank.de“ enters something in your browser, you don\u0026rsquo;t end up on the real online banking portal, but on a phishing page that looks real.\nThis attack is hazardous because it is often difficult to detect. Unlike classic phishing, which relies on fake links or typos, DNS hijacking makes the actual URL appear completely legitimate. The certificate can also appear valid under certain circumstances—for example, with local man-in-the-middle proxies or if the attacker manages to obtain a certificate via compromised certificate authorities.\nJava applications are also potentially affected, primarily if they communicate dynamically with external services. If the host machine\u0026rsquo;s DNS configuration is compromised, all DNS lookups within the JVM are affected, from REST clients to LDAP connections to email components. The application often does not recognize the problem because the connection is technically established correctly, just to the wrong destination.\nThe only solution is to take measures that secure DNS on several levels: through fixed resolver configurations, validation of the other side via TLS, use of secure DNS protocols such as DNS-over-HTTPS (DoH) or DNS-over-TLS (DoT), and by monitoring for suspicious name resolutions. In security-critical Java applications, it is recommended to compare the resolved IP addresses with allowlists or even to forego DNS queries if the target systems are known.\nDNS hijacking is a remarkably sophisticated form of deception because it attacks trust where it seems most evident: in the integrity of the technical infrastructure itself.\n7. Attack detected – but how? # Detecting DNS attacks is one of the most challenging tasks in network security. This is primarily because DNS is a widespread, dynamic and often invisible protocol. Name resolution occurs in the background, usually without direct user contact, and is often only incompletely recorded in logs and monitoring systems. Accordingly, many DNS-based attacks remain undetected for a long time, mainly when they are carried out precisely and subtly.\nA central problem: Neither the browser nor the application registers whether a DNS response was legitimate, as long as the response is formally correct. A manipulated result usually does not differ from a real one, unless it is actively checked. This is where anomaly detection methods come into play. They try to identify unusual behavior, such as sudden changes in the IP assignment of known domains, an accumulation of DNS responses with short TTL or unusual destination addresses outside of usual networks.\nIntrusion detection systems (IDS) and DNS-specific monitoring solutions such as Zeek, OpenDNS or SecurityOnion can help detect such anomalies. However, the prerequisite is that DNS traffic is completely recorded and evaluated, which brings additional complexity with encrypted DNS (DoH/DoT). In controlled networks, it is therefore recommended to route DNS via dedicated resolvers that can be logged and monitored.\nIn Java applications, suspicious name resolutions can be detected by logging IP mappings, comparing against expected hostnames, or active whitelisting. Metrics such as the number of different IP addresses per domain over time or the reuse of certain, unexpected IP ranges are also valuable information.\nIn the long term, comprehensive validation of DNS responses via DNSSEC is essential in ensuring integrity. However, DNSSEC is not retroactive - it only protects when domain owners and resolvers actively work together. Accordingly, awareness of DNS attacks and their detection is a crucial component of any security strategy: Anyone who only sees DNS as a technical infrastructure risks being compromised exactly where trust begins - with name resolution.\n8. Protective measures – think and implement DNS securely # 8.1 General Network Protection Measures # Detecting DNS attacks is one of the most challenging tasks in network security. This is primarily because DNS is a widespread, dynamic and often invisible protocol. Name resolution occurs in the background, usually without direct user contact, and is often only incompletely recorded in logs and monitoring systems. Accordingly, many DNS-based attacks remain undetected for a long time, especially when they are carried out specifically and subtly.\nA central problem: Neither the browser nor the application registers whether a DNS response was legitimate, as long as the reaction is formally correct. A manipulated result usually does not differ from a real one, unless it is actively checked. This is where anomaly detection methods come into play. They try to identify unusual behavior, such as sudden changes in the IP assignment of known domains, an accumulation of DNS responses with short TTL or unusual destination addresses outside of usual networks.\nIntrusion detection systems (IDS) and DNS-specific monitoring solutions such as Zeek, OpenDNS or SecurityOnion can help detect such anomalies. However, the prerequisite is that DNS traffic is completely recorded and evaluated, which brings additional complexity with encrypted DNS (DoH/DoT). In controlled networks, it is therefore recommended to route DNS via dedicated resolvers that can be logged and monitored.#### 8.2 Security within Java applications\nIn Java applications, suspicious name resolutions can be detected by logging IP mappings, comparing against expected hostnames, or active whitelisting. Metrics such as the number of different IP addresses per domain over time or the reuse of certain, unexpected IP ranges are also valuable information.\nIn the long term, comprehensive validation of DNS responses via DNSSEC is essential in ensuring integrity. However, DNSSEC is not retroactive - it only protects when domain owners and resolvers actively work together. Accordingly, awareness of DNS attacks and their detection is a crucial component of any security strategy: Anyone who only sees DNS as a technical infrastructure risks being compromised exactly where trust begins - with name resolution.\n8.3 Advanced protection measures at protocol and application level # Various protection measures can be implemented for Java applications to minimize the risks of DNS-based attacks. A key approach is securing the name resolution process within the JVM. The class InetAddress, for example, uses resolvers close to the operating system, which makes the JVM fundamentally vulnerable to manipulation at the system level. In security-critical applications, it can, therefore, make sense to implement your own resolvers with DNSSEC validation or to use external, verified services.\nAdditionally, it is recommended to be combined with transport encryption and hostname validation at the application level. The HttpClient from the java.net.http module in Java 11+ enables e.g. B. direct control over TLS certificate checking. By securing via public key pinning, certificate transparency logs or targeted whitelists, manipulated DNS responses can be detected and blocked.\nAnother security option is to use alternative DNS protocols such as DoH (DNS over HTTPS) or DoT (DNS over TLS) via upstream proxies. These can also be integrated into containerized environments such as Kubernetes as sidecar resolvers. A locally running DNS proxy (e.g., CoreDNS with DNSSEC and DoH support) can be used as a trustworthy source within the application.\nA practical example of a hedging measure in Java is using your own X509TrustManager to not rely solely on the system-provided chain of trust for the TLS connection. This is particularly useful if you want to counter DNS manipulations that do not break the TLS channel but subvert a false certificate chain - for example, through a compromised CA certificate or a manipulated trust store.\nTrustManager[] trustManagers = new TrustManager[] {\n** new X509TrustManager() {**\n** public X509Certificate[] getAcceptedIssuers() {**\n** return new X509Certificate[0];**\n** }**\n** public void checkClientTrusted(X509Certificate[] certs, String authType) {**\n** // Application-specific testing – e.g. E.g. fingerprinting or pinning**\n** }**\n** public void checkServerTrusted(X509Certificate[] certs, String authType) throws CertificateException {**\n** for (X509Certificate cert : certs) {**\n** cert.checkValidity();**\n** if (!\u0026ldquo;CN=trusted.example.com\u0026rdquo;.equals(cert.getSubjectDN().getName())) {**\n** throw new CertificateException(\u0026ldquo;Untrusted CN: \u0026quot; + cert.getSubjectDN());**\n** }**\n** }**\n** }**\n** }**\n};\nSSLContext sslContext = SSLContext.getInstance(\u0026ldquo;TLS\u0026rdquo;);\nsslContext.init(null, trustManagers, new SecureRandom());\nHttpClient client = HttpClient.newBuilder()\n** .sslContext(sslContext)**\n** .build();**\nThis example deliberately checks the subject of the server certificate passed. This makes it possible to ensure that the remote site actually has the expected identity, even if a DNS hijacker manipulates the target system. A more detailed check would be necessary in a productive system, for example, via public key pinning, dedicated trust stores or certificate chain validation.\nLast but not least: DNS requests should be logged, checked for plausibility and ideally compared with an allowlist. This allows a simple but effective protective layer to be implemented, especially in applications where external communication is security-critical.\n9. Conclusion – trust needs protection # Given the central role of the domain name system in virtually every form of modern network communication, it is remarkable how long DNS security played a secondary role in the architecture of many applications. The forms of attack highlighted here - from cache poisoning to DNS hijacking to tunneling and DDoS - clearly show that DNS is much more than a harmless auxiliary component. It is one of the most critical infrastructures of all, and simultaneously one of the most vulnerable.\nAnyone who develops applications today that rely on Internet communication must think about DNS functionally and in terms of security. This applies to web portals, microservices, container environments, and IoT devices. In the Java ecosystem, in particular, the platform from version 11 provides numerous APIs and mechanisms to specifically secure DNS dependencies, be it through explicit TLS configurations, validation of target addresses, logging of suspicious resolutions, or the use of alternative resolvers.\nDNS is not a static problem. The attack vectors continue to develop, and with them, the requirements for protective mechanisms. At a time when trust is becoming a currency of digital interactions, DNS can no longer remain a blind spot. It is time for DNS security to be established as an integral part of every security-conscious software and infrastructure architecture - in the code, operations and all stakeholders\u0026rsquo; minds. logged, checked for plausibility and ideally compared with an allowlist. This allows a simple but effective protective layer to be implemented, especially in applications where external communication is security-critical.\nHappy Coding\nSven\n","date":"7 April 2025","externalUrl":null,"permalink":"/posts/dns-attacks-explained/","section":"Posts","summary":"1. Getting started – trust in everyday internet life # Anyone who enters a web address like “www.example.de” into the browser expects a familiar website to appear within seconds. Whether in the home office, at the university, or in the data center, access to online services is now a given. The underlying technical processes are invisible to most users; even in IT practice, they are often taken for granted. One of these invisible processes is name resolution by the Domain Name System (DNS).\n","title":"DNS Attacks - Explained","type":"posts"},{"content":"","date":"3 April 2025","externalUrl":null,"permalink":"/tags/cryptography/","section":"Tags","summary":"","title":"Cryptography","type":"tags"},{"content":"The Java Cryptography Architecture (JCA) is an essential framework within the Java platform that provides developers with a flexible and extensible interface for cryptographic operations. It is a central component of the Java Security API and enables platform-independent implementation of security-critical functions.\nAt its core, the JCA provides mechanisms for various cryptographic applications, including the calculation of hashes to ensure data integrity, the generation and verification of digital signatures, and methods for encrypting and decrypting sensitive information. Supporting both symmetrical and asymmetrical encryption methods, it ensures a high level of security when processing data. Cryptographic key management is another key aspect that includes secure storage and exchange of keys.\nHow is the JCA integrated into the Security API? Symmetric encryption The first practical steps Asymmetric encryption A practical example Digital signatures A practical example Now let\u0026rsquo;s combine symmetric and asymmetric encryption A significant feature of the JCA is the ability to integrate different security providers (so-called providers) that implement different cryptographic algorithms. In addition to the providers that come standard with Java, such as SunJCE, there are alternative implementations, including open source libraries such as Bouncy Castle and hardware-based solutions, that can be integrated via PKCS#11 interfaces.\nThe JCA\u0026rsquo;s architecture is characterised by a modular design that allows a separation between the API and the concrete implementations. New cryptographic algorithms can be added via the so-called Service Provider Interface (SPI) without having to adapt existing application code, contributing to the long-term maintainability and security of Java applications.\nIn addition to classic cryptography, the JCA also supports mechanisms for the secure generation of random numbers, the calculation of Message Authentication Codes (MACs) to ensure data integrity, and the management of certificates and trust stores. In particular, integration with the Java KeyStore API offers a comprehensive basis for the secure handling of X.509 certificates and cryptographic keys.\nHow is the JCA integrated into the Security API? # The Java Security API is a comprehensive framework for implementing security-related mechanisms in Java applications. It provides various functions that cover various aspects of IT security, including cryptography, authentication, authorization, network security, and cryptographic key management. A central component of this framework is the Java Cryptography Architecture (JCA).\nWhile JCA is primarily responsible for the cryptographic functions, it extends the Java Cryptography Extension (JCE). This concept includes algorithms for symmetric encryption, key agreement and block cipher modes. In addition to cryptography, the Java Security API includes other security-relevant modules. The Java Secure Socket Extension (JSSE) API enables secure communication via network protocols such as TLS and SSL, enabling encrypted connections for web applications and other network services to be realised. This provides authentication and authorisation Java Authentication and Authorization Service (JAAS) API mechanisms available to verify user identities and control access rights. This architecture allows integration with various authentication sources such as user-password combinations, Kerberos or biometric methods.\nAnother essential component of the framework is the Java Public Key Infrastructure (PKI) API , which deals with the management of digital certificates and public keys. This API plays a central role in the realization of TLS encryption and the implementation of digital signatures. Certificate management is carried out via so-called KeyStores, which enable secure storage of cryptographic keys. In addition, the framework supports digital signatures for XML documents through the Java XML Digital Signature API (XMLSig) , which is particularly used in web services and SAML authentication scenarios.\nThe distinction between the Java Security API and the JCA lies in their respective focus. While the Java Security API provides a comprehensive collection of security-related mechanisms and covers both cryptographic and non-cryptographic security aspects such as authentication, permission management and network security, the JCA focuses exclusively on providing cryptographic operations. In this context, JCA offers basic mechanisms for encryption and digital signatures, but JCE is required for modern symmetric encryption methods. Likewise, although JCA includes basic functions for managing cryptographic keys, more comprehensive management of certificates and public key infrastructure requires the use of the PKI API.\nIn practice, this means that developers access the JCA and JCE directly to implement cryptographic security mechanisms, while specific modules such as JAAS or JSSE are used for higher security-related functions such as authentication, authorisation or the management of secure network connections. The Java Security API thus forms a coherent but modular security architecture that can be used specifically depending on the application.\nIn this article we will only deal with the JCA. The other functional units will be gradually added in later articles.\nSymmetric encryption # The symmetric encryption within the Java Cryptography Architecture (JCA) offers a powerful and flexible interface for the secure processing of confidential data. It is based on the use of a single secret key for encryption and decryption, which makes it more efficient and faster than asymmetric methods. The JCA provides the Cipher-Class that acts as a central mechanism for symmetric cryptography and enables the implementation of various block and stream ciphers.\nA key feature of symmetric encryption in the JCA is its support for modern block ciphers such as AES (Advanced Encryption Standard) , which is considered the current industry standard, as well as older methods such as DES (Data Encryption Standard) and its improved version, Triple DES (3DES). The API allows the use of various operating modes, including ECB (Electronic Codebook) , which is considered unsafe, as well as safer modes such as CBC (Cipher Block Chaining) , CFB (Cipher Feedback Mode) and GCM (Galois/Counter Mode). GCM in particular is preferred because, in addition to confidentiality, it also offers integrity protection through Authenticated Encryption (AEAD).\nA central concept of the JCA is the separation between the abstract API and the concrete implementations provided by various cryptography providers. This keeps the actual implementation interchangeable without tying the application to a specific algorithm or library. The initialization of a Cipher-Object is done by specifying the algorithm as well as the desired operating mode and an optional padding method that ensures block-by-block processing of the data. Common padding schemes are PKCS5Padding and NoPadding , the latter requiring manual adjustment of the input data.\nThe JCA also provides mechanisms for the secure generation and management of keys. About the KeyGenerator-Class, symmetrical keys can be generated based on a defined key length, with key lengths of 128, 192 and 256 bits being particularly common for AES. To ensure the security of the keys, they can be passed through the Java KeyStore (JKS) or external hardware security modules (HSMs). Additionally, the API allows keys to be imported and exported in standardised formats to ensure interoperability with other cryptographic systems.\nAnother central element of symmetric encryption in the JCA is the treatment of initialisation vectors (IVs), which are crucial for the security of the encryption, especially in modes such as CBC or GCM. IVs must be unique for each encryption operation to prevent attacks such as replay or chosen plaintext attacks. In practice, IVs are often over SecureRandom generated to ensure high entropy and stored or transmitted along with the encrypted data.\nThe architecture of the JCA allows developers to use symmetric encryption not only for straightforward file and message encryption, but also in more complex scenarios such as Transport Layer Security (TLS) , Database encryption and Hardware-assisted encryption. By combining it with other JCA components, such as the Mac class for Message Authentication Codes (e.g. HMAC-SHA256) or Key Derivation Functions (e.g. PBKDF2) , additional security features such as authentication and key derivation can be implemented.\nThe first practical steps # Let\u0026rsquo;s now move on to a simple example of using symmetric encryption within the Java Cryptography Architecture (JCA) and use the AES algorithm family in combination with a secure operating mode and padding. The implementation relies on the Cipher-Class, a central interface in Java for processing cryptographic operations. Choosing secure key management is crucial, which is why the key is stored using the KeyGenerator class, which is created. SecureRandom is used to ensure high entropy and encryption security for the initialisation data.\nWithin the method, a character array (char[]) is used instead of a string to ensure that sensitive data does not remain in the string pool and can be specifically overwritten after processing. (Here, I refer to the previous article, in which I described this in more detail.) After the key and IV initialisation, encryption is carried out using a Cypher object in AES-CBC mode with PKCS5Padding. The encrypted string is then stored in a byte array. Decryption is done in reverse order by reinitialising the Cipherobject with the same key and IV, which allows the original characters to be restored from the encrypted data stream.\nThe following source code demonstrates the implementation of this concept:\nimport javax.crypto.Cipher; import javax.crypto.KeyGenerator; import javax.crypto.SecretKey; import javax.crypto.spec.IvParameterSpec; import java.security.SecureRandom; import java.util.Arrays; public class SymmetricEncryptionExample { public static void main(String[] args) throws Exception { KeyGenerator keyGenerator = KeyGenerator.getInstance(\u0026#34;AES\u0026#34;); keyGenerator.init(256); SecretKey secretKey = keyGenerator.generateKey(); SecureRandom secureRandom = new SecureRandom(); byte[] iv = new byte[16]; secureRandom.nextBytes(iv); IvParameterSpec ivSpec = new IvParameterSpec(iv); char[] originalText = {\u0026#39;H\u0026#39;, \u0026#39;e\u0026#39;, ​​\u0026#39;l\u0026#39;, \u0026#39;l\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39; \u0026#39;, \u0026#39;J\u0026#39;, \u0026#39;C\u0026#39;, \u0026#39;A\u0026#39;}; byte[] encryptedText = encrypt(originalText, secretKey, ivSpec); char[] decryptedText = decrypt(encryptedText, secretKey, ivSpec); System.out.println(decryptedText); Arrays.fill(decryptedText, \u0026#39;\\0\u0026#39;); } private static byte[] encrypt(char[] input, SecretKey key, IvParameterSpec ivSpec) throws Exception { Cipher cipher = Cipher.getInstance(\u0026#34;AES/CBC/PKCS5Padding\u0026#34;); cipher.init(Cipher.ENCRYPT_MODE, key, ivSpec); return cipher.doFinal(new String(input).getBytes()); } private static char[] decrypt(byte[] input, SecretKey key, IvParameterSpec ivSpec) throws Exception { Cipher cipher = Cipher.getInstance(\u0026#34;AES/CBC/PKCS5Padding\u0026#34;); cipher.init(Cipher.DECRYPT_MODE, key, ivSpec); byte[] decryptedBytes = cipher.doFinal(input); char[] decryptedChars = new char[decryptedBytes.length]; for (int i = 0; i \u0026lt; decryptedBytes.length; i++) { decryptedChars[i] = (char) decryptedBytes[i]; } return decryptedChars; } } Asymmetric encryption # The asymmetric encryption within the Java Cryptography Architecture (JCA) provides a flexible and secure way to encrypt and decrypt data using two different keys: a public key for encryption and a private key for decryption. This method is based on one-way mathematical functions, which make it practically impossible to calculate the private key from the public key. This makes asymmetric cryptography particularly suitable for applications that require secure key distribution and digital signatures, including Transport Layer Security (TLS), Public Key Infrastructure (PKI), and various authentication mechanisms.\nThe JCA provides with the Cipherclass provides a central API that enables the implementation of asymmetric encryption methods. The algorithms are particularly widespread RSA (Rivest-Shamir-Adleman) , Elliptic Curve Cryptography (ECC) as well as Diffie-Hellman for key exchange. RSA is based on the factorization of large prime numbers and is widely used as a classic public key method. In contrast, ECC uses elliptic curves over finite fields and offers comparable security with a smaller key length, making it particularly preferred for resource-constrained environments such as mobile devices or smart cards.\nThe key is generated via the KeyPairGenerator-Class that allows to generate asymmetric key pairs with different key lengths. The choice of key length affects security and computing power, with RSA typically operating at 2048 or 4096 bits, while elliptic curves use more efficient 256 or 384 bit keys. The public key generated can be freely distributed, while the private key must be kept secure, for example by storing it in a Java KeyStore (JKS) or a hardware security solution such as a Hardware Security Module (HSM).\nA Cipher object is initialised with the desired algorithm and mode for the encryption. The JCA supports various padding mechanisms necessary to adapt input data to the block size of the encryption function used. At RSA, PKCS1Padding and OAEP (Optimal Asymmetric Encryption Padding) are standard methods, with OAEP being preferred due to its increased security, as it protects against adaptive chosen ciphertext attacks. Due to its high computational complexity, asymmetric encryption is primarily suitable for encrypting small amounts of data, such as key material or hash values, rather than large files or streaming data.\nAnother central application area of ​​asymmetric cryptography in the JCA is the digital signature, which is carried out via the Signature class. The private key creates a signature for a message or file, which can later be verified using the public key. Digital signatures ensure both the authenticity and integrity of the transmitted data by ensuring that the message comes from the specified sender and has not been subsequently altered. In particular, algorithms like RSA with SHA-256 , ECDSA (Elliptic Curve Digital Signature Algorithm) or EdDSA (Edwards-Curve Digital Signature Algorithm) are used in modern applications to ensure authenticity and data integrity.\nIn addition to classic asymmetric encryption, the JCA also supports hybrid methods in which asymmetric cryptography is used to securely transmit a symmetric key, which is then used for data encryption. This procedure is used, for example, TLS (Transport Layer Security) or PGP (Pretty Good Privacy), to combine the advantages of both types of cryptography: the security of asymmetric key distribution and the efficiency of symmetric encryption.\nThe JCA\u0026rsquo;s architecture allows developers to flexibly use asymmetric encryption schemes by supporting a variety of security vendors that provide different implementations. By default, the Java platform offers support for RSA, DSA and elliptic curves, while alternative providers such as Bouncy Castle enable additional algorithms and optimisations.\nA practical example # This example uses the RSA algorithm to implement asymmetric encryption, encrypting data with a public key and decrypting it with the corresponding private key. The implementation is done using the Cipher class. The KeyPairGenerator class is used to generate the key pair, which makes it possible to generate an RSA key with a length of 2048 bits. The resulting key pair consists of a public key used for encryption and a private key required for decryption.\nThe encryption method takes char[] as input to ensure that no sensitive data is included as String remains in memory. The characters are explicitly converted to a byte array before encryption to enable direct storage operations. Afterwards, Cipherobject is put into encryption mode with the public key, and the conversion is performed. Once encryption is complete, the original byte array is securely overwritten to prevent potential reconstruction in memory.\nDecryption is carried out using an analogous procedure. The Cipher object is initialised with the private key to return the previously encrypted byte array to its original state. After decryption, the byte array turns back into a char[] converted without doing a String conversion. To protect the decrypted data, the original and decrypted byte arrays are overwritten with null values ​​after processing.\nimport javax.crypto.Cipher; import java.security.KeyPair; import java.security.KeyPairGenerator; import java.security.PrivateKey; import java.security.PublicKey; import java.util.Arrays; public class AsymmetricEncryptionExample { public static void main(String[] args) throws Exception { KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance(\u0026#34;RSA\u0026#34;); keyPairGenerator.initialize(2048); KeyPair keyPair = keyPairGenerator.generateKeyPair(); PublicKey publicKey = keyPair.getPublic(); PrivateKey privateKey = keyPair.getPrivate(); char[] originalText = {\u0026#39;H\u0026#39;, \u0026#39;e\u0026#39;, ​​\u0026#39;l\u0026#39;, \u0026#39;l\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39; \u0026#39;, \u0026#39;J\u0026#39;, \u0026#39;C\u0026#39;, \u0026#39;A\u0026#39;}; byte[] encryptedText = encrypt(originalText, publicKey); char[] decryptedText = decrypt(encryptedText, privateKey); System.out.println(decryptedText); Arrays.fill(decryptedText, \u0026#39;\\0\u0026#39;); Arrays.fill(originalText, \u0026#39;\\0\u0026#39;); } private static byte[] encrypt(char[] input, PublicKey key) throws Exception { Cipher cipher = Cipher.getInstance(\u0026#34;RSA/ECB/OAEPWithSHA-256AndMGF1Padding\u0026#34;); cipher.init(Cipher.ENCRYPT_MODE, key); byte[] inputBytes = new byte[input.length]; for (int i = 0; i \u0026lt; input.length; i++) { inputBytes[i] = (byte) input[i]; } byte[] encryptedBytes = cipher.doFinal(inputBytes); Arrays.fill(inputBytes, (byte) 0); return encryptedBytes; } private static char[] decrypt(byte[] input, PrivateKey key) throws Exception { Cipher cipher = Cipher.getInstance(\u0026#34;RSA/ECB/OAEPWithSHA-256AndMGF1Padding\u0026#34;); cipher.init(Cipher.DECRYPT_MODE, key); byte[] decryptedBytes = cipher.doFinal(input); char[] decryptedChars = new char[decryptedBytes.length]; for (int i = 0; i \u0026lt; decryptedBytes.length; i++) { decryptedChars[i] = (char) decryptedBytes[i]; } Arrays.fill(decryptedBytes, (byte) 0); return decryptedChars; } } Digital signatures # Digital signatures are cryptographic mechanisms that ensure digital messages or documents\u0026rsquo; authenticity, integrity and non-repudiation. They are based on asymmetric cryptography and use a key pair consisting of a private key used to generate the signature and a public key used for verification. The central idea behind digital signatures is to confirm the sender\u0026rsquo;s identity and ensure that the transmitted data has not been tampered with during transmission or storage.\nCreating a digital signature begins with calculating a hash value of the message to be signed using a cryptographic hash function. This hash value represents a unique, compact representation of the original message, with even a minimal change to the message resulting in a completely different hash. This hash value is then encrypted with the sender\u0026rsquo;s private key, creating the digital signature. This signature is transmitted or stored along with the original message to ensure its authenticity later.\nThe recipient verifies the digital signature using the sender\u0026rsquo;s public key. To do this, the received message is first hashed again to calculate a new hash value. In parallel, the digital signature is decrypted using the public key, restoring the original hash value generated by the sender. If both hash values ​​match, the recipient can assume with a high degree of certainty that the message is authentic and has not been changed. If the hash values ​​are different, it means either that the message has been tampered with or that the signature was created with a different key, indicating an unauthorised modification or a forged identity.\nThe security of digital signatures depends largely on the strength of the underlying mathematical procedures and the secure handling of the private keys. Classic signature algorithms like RSA or DSA (Digital Signature Algorithm) are increasingly being used by more modern processes such as ECDSA (Elliptic Curve Digital Signature Algorithm) or EdDSA (Edwards-Curve Digital Signature Algorithm) , as they offer comparable security with shorter keys and higher efficiency. The continuous development of cryptographic standards is essential to keep digital signatures secure in the long term against future threats, such as those from quantum computers.\nA practical example # This digital signature implementation example demonstrates the creation and verification of a signature using the RSA signature algorithm with SHA-256. The Signature-Class carries out the implementation. To generate a valid key pair, the KeyPairGenerator class makes it possible to generate an RSA key pair with a length of 2048 bits. The resulting key pair consists of a private key, which is used to create the signature, and a public key, which is used for later verification.\nSignature generation begins by processing an input known as char[] that is present. Before signing, the characters are converted directly into a byte array without creating strings. The Signature instance is initialised with the private key, after which the data is processed and the signature is generated. This is stored in a separate byte array, while the original input array is overwritten with null values ​​after processing to minimise security risks.\nThe signature is verified by initialising the Signature instance again, this time with the public key. The received data is passed to signature verification in byte form, ensuring that it matches the originally signed data. If the verification is successful, it means that the data has remained unchanged and comes from the specified source. On the other hand, a failure of verification indicates that either the data or the signature was manipulated or that an incorrect public key was used for verification.\nThe following source code shows an example usage:\nimport java.security.KeyPair; import java.security.KeyPairGenerator; import java.security.PrivateKey; import java.security.PublicKey; import java.security.Signature; import java.util.Arrays; public class DigitalSignatureExample { public static void main(String[] args) throws Exception { KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance(\u0026#34;RSA\u0026#34;); keyPairGenerator.initialize(2048); KeyPair keyPair = keyPairGenerator.generateKeyPair(); PublicKey publicKey = keyPair.getPublic(); PrivateKey privateKey = keyPair.getPrivate(); char[] message = {\u0026#39;H\u0026#39;, \u0026#39;e\u0026#39;, \u0026#39;l\u0026#39;, \u0026#39;l\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39; \u0026#39;, \u0026#39;J\u0026#39;, \u0026#39;C\u0026#39;, \u0026#39;A\u0026#39;}; byte[] signature = signData(message, privateKey); boolean isValid = verifyData(message, signature, publicKey); System.out.println(\u0026#34;Signature valid: \u0026#34; + isValid); Arrays.fill(message, \u0026#39;\\0\u0026#39;); } private static byte[] signData(char[] input, PrivateKey key) throws Exception { Signature signature = Signature.getInstance(\u0026#34;SHA256withRSA\u0026#34;); signature.initSign(key); byte[] inputBytes = new byte[input.length]; for (int i = 0; i \u0026lt; input.length; i++) { inputBytes[i] = (byte) input[i]; } signature.update(inputBytes); byte[] signedBytes = signature.sign(); Arrays.fill(inputBytes, (byte) 0); return signedBytes; } private static boolean verifyData(char[] input, byte[] signatureBytes, PublicKey key) throws Exception { Signature signature = Signature.getInstance(\u0026#34;SHA256withRSA\u0026#34;); signature.initVerify(key); byte[] inputBytes = new byte[input.length]; for (int i = 0; i \u0026lt; input.length; i++) { inputBytes[i] = (byte) input[i]; } signature.update(inputBytes); boolean isValid = signature.verify(signatureBytes); Arrays.fill(inputBytes, (byte) 0); return isValid; } } Now let\u0026rsquo;s combine symmetric and asymmetric encryption # Secure communication between a client and a server can be achieved by combining asymmetric cryptography for key exchange and symmetrical encryption for the actual message transmission. In this simulation, a hybrid encryption system is implemented that initially uses RSA to enable secure transmission of a session key, which is then used for encrypted communication using AES. This architecture is comparable to the handshake process in TLS (Transport Layer Security) , whereby asymmetric cryptography is only used for the initial key exchange. At the same time, the more efficient symmetric encryption secures the messages.\nFirst, the server generates an RSA key pair , which is used to encrypt and decrypt the session key. The client provides the public key while the private key remains on the server. The client then creates one random AES session key , which is encrypted with and sent to the server\u0026rsquo;s RSA public key. Once received, the server decrypts the session key using its private key. Both parties now have the same symmetrical key used for subsequent encrypted communication.\nThe message transmission takes place using the AES algorithm in Galois/Counter Mode (GCM) , which, in addition to confidentiality, also ensures integrity protection through integrated authentication. Each message packet sent is given a random initialisation vector (IV) before encryption to ensure that identical messages do not result in identical ciphertexts.\nIf you remember the login process, you can use this procedure to secure the components transmitted over the network, such as hash values ​​and salt values. At this point, I must, of course, point out again that this is a presentation of the basic principles. For productive use, it is recommended to use existing implementations and protocols.\nHappy Coding\nSven\n","date":"3 April 2025","externalUrl":null,"permalink":"/posts/java-cryptography-architecture-jca-an-overview/","section":"Posts","summary":"The Java Cryptography Architecture (JCA) is an essential framework within the Java platform that provides developers with a flexible and extensible interface for cryptographic operations. It is a central component of the Java Security API and enables platform-independent implementation of security-critical functions.\n","title":"Java Cryptography Architecture (JCA) - An Overview","type":"posts"},{"content":"","date":"3 April 2025","externalUrl":null,"permalink":"/tags/jca/","section":"Tags","summary":"","title":"Jca","type":"tags"},{"content":"","date":"2 April 2025","externalUrl":null,"permalink":"/categories/concurrency/","section":"Categories","summary":"","title":"Concurrency","type":"categories"},{"content":"","date":"2 April 2025","externalUrl":null,"permalink":"/tags/gatherer/","section":"Tags","summary":"","title":"Gatherer","type":"tags"},{"content":"Since version 8, Java has introduced an elegant, functional approach to processing data sets with the Streams API. The terminal operation collect(\u0026hellip;) represents the bridge from the stream to a targeted aggregation - in the form of lists, maps, strings or more complex data structures. Until Java 20 the processing was done Collector-Instances were regulated, which internally consisted of a supplier, an accumulator, a combiner and optionally a finisher. This model works well for simple accumulations but has noticeable limitations, particularly for complex, stateful, or conditional aggregations.\nThe semantics of gatherers Why Gatherers are more than just a “better collector” Integration with Streams API Sequentieller Gatherer Initialisation: The state of the gatherer Accumulation: Processing the input elements Emission: The control over the output Signature and purpose of the finisher Concrete example: chunking with remainder Interaction with accumulation No return – but effect through push() Parallel Gatherer Initializer – The creation of the accumulator Integrator – The processing of elements Combiner – The combination of partial accumulators Finisher – The transformation of the accumulator into the final result Interaction in parallel gatherers An example implementation Initialiser: Integrator: Combiner: entries state1 insert entries state2 insert Finisher: Java 21 was the new interface java.util.stream. The gatherer was introduced, significantly expanding the semantics and control over the accumulation process. A Collector passively collects data, acts as a Gatherer, and actively responds to the incoming elements, which is comparable to a specialised transducer in functional programming languages. Gatherers are particularly useful where procedural or stateful aggregation is necessary, and they also allow element insertion, filtering, skipping, and explicit termination of the gathering process - all within the framework of a functional composable architecture.\nThe semantics of gatherers # A Gatherer\u0026lt;T, R\u0026gt; describes the transformation of one Stream into a result of type R under close control over the accumulation process. In contrast to Collector, which is in a sense a container for aggregation logic, the gatherer allows rule-based, state-dependent processing of inputs - including the ability to skip elements (Drop), to insert additionally (Inject) or to end processing early (FinishEarly).\nTo make this possible, a gatherer is based on the idea of ​​a Sink , which is called in the context of the stream processor. This sink receives every input element, can react to it and thus actively influences the flow of processing. The actual processing is done via a so-called Adapter Factory which manages the transitions between the aggregation states.\nWhy Gatherers are more than just a “better collector” # While the conventional Collector serves primarily as a final accumulation tool - i.e. to transfer the elements contained in the stream into a target structure such as a list, a map or an aggregation - it works Gatherer conceptually far beyond this role. It is not just an optimised or syntactically varied form of Collectors, but rather an independent mechanism that opens up new expressions for stream processing, both semantically and functionally.\nThe central difference lies in the expanded scope for action during the transformation: This means explicitly that a Gatherer can accumulate not only elements but also new, previously non-existent elements that can be fed into the data stream. This opens up the possibility, for example, of introducing initialisation values ​​at the beginning of a stream or placing control characters such as headers and footers specifically at the beginning or end - without artificially expanding the original data stream.\nThis creative freedom becomes particularly clear when dealing with conditions. Where a Collector usually operated with a simple accumulator, the state of which leads to a final result, can be a Gatherer work based on state – and allow this state to be influenced across several elements. This opens up new semantic horizons: For example, window operations can be formulated in which temporal or sequential logic is applied - such as aggregating data up to an inevitable \u0026ldquo;end\u0026rdquo; marker, or combining groups of elements that can only be identified by a particular order or content structure.\nEven complex decision structures, such as those required in multi-stage parsing processes or when traversing decision trees, can be achieved using stateful ones that Gatherer implements elegantly and declaratively. The interface remains in the spirit of functional programming: transformation and aggregation can still be described separately, but the Gatherer ensures their connection in a way that was previously only possible through imperative or difficult-to-maintain stream hacks.\nAnother advantage is the controlled influence of past elements on current behavior. This is how one can Gatherer For example, making the decision to discard an element because a previous element set a certain context. This context sensitivity capability is particularly relevant in situations where data streams are structurally “not clean” – such as log files, inconsistent data exports, or natural language analysis.\nA Gatherer is not just “better Collector\u0026quot;, but a fundamental new tool for data stream-based modeling of complex transformation logic. It opens up a design space in which state, transformation, context and accumulation can interact in a way that was previously only possible with considerable effort - or outside of the stream model. Anyone who has ever worked with stateful Gathererconstructions will notice how much this expands the expressiveness of functional stream architectures.\nA concrete example: grouping with filter logic\nLet\u0026rsquo;s imagine that we want to collect from a stream of strings only those elements that have a particular property and then group them - for example, all words longer than 5 characters, grouped by their first letter. This requirement can be met with a Collector formulate, but requires a combination of preprocessing (e.g. filter(\u0026hellip;)) and downstream grouping. With a Gatherer On the other hand, this combined process can be represented elegantly, comprehensively and in one step:\nGatherer\u0026lt;String, ?, Map\u0026lt;Character, List\u0026lt;String\u0026gt;\u0026gt;\u0026gt; gatherer = Gatherer.ofSequential( () -\u0026gt; new HashMap\u0026lt;Character, List\u0026lt;String\u0026gt;\u0026gt;(), (map, element, downstream) -\u0026gt; { if (element.length() \u0026gt; 5) { char key = element.charAt(0); map.computeIfAbsent( key, k -\u0026gt; new ArrayList\u0026lt;\u0026gt;()).add(element); } return true; } ); In this example, a decision is made for each element as to whether it will be included in the result. The logic is embedded directly into the Gatherer. The return value accurate signals that processing should continue. You would, at this point, instead of false return, and the stream would end prematurely - a behaviour that is not possible with conventional Collectors is not reachable like that.\nIntegration with Streams API # The interface Gatherer\u0026lt;T, A, R\u0026gt; explicitly distinguishes between more sequential and parelleler processing. The central distinction arises from the factory methods:\nGatherer.ofSequential(\u0026hellip;) // Can only be used sequentially\nGatherer.ofConcurrent(\u0026hellip;) // Suitable for parallel streams\nA gatherer who comes with ofConcurrent(\u0026hellip;) may be used in parallel streams, but must meet certain requirements: it must be thread-safe or rely on thread-isolated accumulators. This is similar to the logic of parallel collectors, where internal state management allows different elements to be processed simultaneously in independent threads.\nSequentieller Gatherer # Especially at sequential processing —i.e., if there is no parallelisation—the Gatherer develops its full expressiveness while remaining simple, type-safe, and deterministic.\nThe functionality of a sequential gatherer can be divided into three main phases: initialisation , accumulation and Emission. Each of these phases is described in detail below, with particular attention to the unique features of sequential processing.\nInitialisation: The state of the gatherer # Each gatherer has an internal state that is recreated per stream execution. This condition is about one Supplier defined, where S represents the type of condition. In sequential processing, this state is reserved exclusively for a single thread; therefore, nothread safety requirements exist. This means that simple objects like ArrayList, StringBuilder, counter arrays, and custom records can be used without problems.\nA typical condition could e.g. B. look like this:\nrecord ChunkState (List buffer, int chunkSize) {}\nThe associated supplier:\n() - \u0026gt; new ChunkState\u0026lt;\u0026gt;(new ArrayList\u0026lt;\u0026gt;(chunkSize), chunkSize)\nThe state lives for the entire stream run and serves as context for all further processing steps. Initialisation lays the foundation for state-based logic, such as buffering, counting, aggregation or tracking previous elements.\nAccumulation: Processing the input elements # The accumulation function is the heart of every gatherer. It is called for each element of the input stream. The signature of this function is:\n(state, input, downstream) - \u0026gt; { \u0026hellip; }\nThis is where the actual logic happens: The input element is brought into the state, and - depending on the current state - a decision can be made as to whether (and if necessary, how many) output values ​​are produced. The decision as to whether an item is passed downstream rests entirely with the gatherer.\nExample: Every third element of a stream should be emitted.\nGatherer\u0026lt;String, ?, String\u0026gt; everyThird() { return Gatherer.ofSequential( () -\u0026gt; new int[]{0}, (state, element, downstream) -\u0026gt; { state[0]++; if (state[0] % 3 == 0) { downstream.push(element); } } ); } In contrast to classic filter or map operations, this logic is conditional and imperative : The gatherer remembers how many elements have already been processed and only emits a result for every third. The accumulation logic is, therefore, comparable to that of the accept () Method of a specially defined consumer, but supplemented by downstream control.\nSince there is no threading in sequential processing, all operations can be performed directly without synchronisation. The state can be arbitrarily complex and dynamic as long as it is updated correctly within the stream.\nEmission: The control over the output # Elements are output via the Sink-Object provided to the Gatherer upon each accumulation. With his method, push(R element) elements can be passed downstream in a targeted and controlled manner. Instead of map or flatMap, where each input leads to one or more outputs automatically transformed, the gatherer decides himself, if , at and was he emits.\nFor example, a gatherer can:\npush individual output values, push multiple values ​​at once (e.g. with chunking or tokenisation), completely suppress the emission (e.g. under preconditions), generate values ​​with a delay (e.g., at the stream\u0026rsquo;s end or after accumulation thresholds). Example: Combining three consecutive elements into a string:\nGatherer\u0026lt;String, ?, String\u0026gt; triplets() { return Gatherer.ofSequential( () -\u0026gt; new ArrayList\u0026lt;String\u0026gt;(3), (state, element, downstream) -\u0026gt; { state.add(element); if (state.size() == 3) { downstream.push(String.join(\u0026#34;-\u0026#34;, state)); state.clear(); } }); } The emission here only occurs when three elements have been collected. These are merged, pushed and the state is then emptied.\nAn often overlooked but essential component of a sequential gatherer is the Finisher – i.e. the closing function after the last input element. This phase is crucial because often during regular accumulation Items retained in state which will only be done at a later date or even no longer through regular accumulation can be emitted. The finisher ensures that such remaining elements or aggregated partial results are not lost but are correctly transferred downstream.\nSignature and purpose of the finisher # The closing function has the signature:\nBiConsumer \u0026lt;State, Sink\u0026gt;\nshe will after all input values ​​have been processed called by the stream framework – exactly once. In this function, the final state can be accessed and, based on this state, a decision can be made as to whether (additional) output values ​​should be created.\nThe finisher is particularly suitable for:\nPartially filled buffers , for example in chunking operations when the last block does not reach full size, Final aggregations , e.g. B. in averaging, summation, hash calculation or protocol completion, Finalisation of state machines , e.g. B. if an open state still needs to be completed, Cleaning or logging , e.g. B. statistical outputs or final indicators. Concrete example: chunking with remainder # Let\u0026rsquo;s look again at the example of a gatherer that groups elements into groups of three. Without finishers, if the number of elements is odd, the last one or two values ​​would be lost:\nGatherer\u0026lt;String, ?, String\u0026gt; triplets() { return Gatherer.ofSequential( () -\u0026gt; new ArrayList\u0026lt;String\u0026gt;(3), (state, element, downstream) -\u0026gt; { state.add(element); if (state.size() == 3) { downstream.push(String.join(\u0026#34;-\u0026#34;, state)); state.clear(); } }, (state, downstream) -\u0026gt; { if (!state.isEmpty()) { downstream.push(String.join(\u0026#34;-\u0026#34;, state)); } } ); } In the closing function (finish), it is explicitly checked whether there are still elements in the state—i.e., whether the buffer is incomplete. These residual values ​​are then combined into an aggregate and pushed.\nWithout the finisher there would be the gatherer functionally incomplete : For input sets that are not a multiple of three, the last chunk would simply be discarded - a classic off-by-one error.\nInteraction with accumulation # The finisher is semantically separated from the accumulation logic, but accesses the same state. This means that, depending on the specific application, it can use the same auxiliary functions or serialisation routines as the accumulation itself. In practice, it is advisable to define auxiliary methods for common logic such as \u0026ldquo;combine and empty the list\u0026rdquo; in order to avoid redundancy.\nNo return – but effect through push() # The finisher gives no return value back, but - like the accumulation function - works via what is provided Sink. So it doesn\u0026rsquo;t find any return semantics, instead a controlled completion of processing push() views.\nThe finisher of a sequential gatherer is the binding conclusion of the processing model. He guarantees that all information remaining in the state is processed and, if necessary, emitted. Especially in data stream-based applications where incomplete blocks, open ends, or residual states are typical, the finisher is essential to avoid data loss and ensure semantic correctness. Therefore, the finisher has a clean gatherer design that is not optional but rather an integral part of a well-defined stream processing step.\nA sequential gatherer combines:\nthe state handling of an aggregator, the control flow options of a parser, the expressiveness of an imperative processor, and the clarity of functional APIs. By foregoing parallelisation logic and concurrency, the sequential variant allows a gatherer to be developed with minimal overhead and maximum expressiveness - a tool that combines both the flexibility of imperative logic and the composition of functional programming.\nParallel Gatherer # A parallel gatherer is for parallel Data processing pipelines are responsible for the four phases initialiser, integrator, combiner and finisher can be explicitly separated and controlled from each other.\nInitializer – The creation of the accumulator # The method initialiser defines how a new accumulator (internal state) is created. This is the first step in processing each substream in sequential and parallel pipelines.\nThe signature is also: Supplier initializer();\nIn parallel processing, this initialiser is called several times - once per substream, i.e. per thread that takes over a split of the data. This ensures that no synchronisation within the accumulator is necessary: ​​each thread operates in isolation with its own state.\nIntegrator – The processing of elements # The integrator is a central function for inserting stream elements into the accumulator. It is one BiConsumer\u0026lt;A, T\u0026gt;, i.e. a function for each element T the Accumulator A changed accordingly.\nThe signature reads: BiConsumer \u0026lt;A, ? super T\u0026gt; integrator();\nIn parallel streams, this integrator is also applied to partial accumulators. What is important here is that this function may only change the accumulator locally and may not influence any global states.\nCombiner – The combination of partial accumulators # The combiner\u0026rsquo;s task is to combine several independently processed accumulators into one overall accumulator. This phase is only relevant in parallel stream executions. The combiner receives two partial results—typically from two threads—and has to combine them into a common result.\nThe signature is: BinaryOperator combiner();\nThe correct implementation of the combiner is essential for the correctness of the parallel execution. It must be associative. This is the only way the JVM can freely distribute the operation across multiple threads.\nFinisher – The transformation of the accumulator into the final result # The finisherfunction transforms the accumulator A into the desired result R. While A is used internally to work efficiently during aggregation R the actual result of the entire operation - such as an immutable collection, a merged string, an optional, a report object, etc.\nThe signature reads: Function \u0026lt;A, R\u0026gt; finisher();\nUnlike the integrator and combiner, the finisher becomes accurate once called, at the end of the entire processing chain. It therefore serves as a bridge between the internal aggregation mechanism and external representation.\nInteraction in parallel gatherers # In a parallel stream with gatherer-based collector, the following happens:\nAn initialiser is called for each partial stream (split) to create a local accumulator. The integrator processes all elements in the respective substream one after the other and modifies the local accumulator. The combiner phase is called as soon as two partial accumulators need to be combined. This happens either through ForkJoin merging or in the final reduction step. After all partial accumulators have been combined, the finisher is called to calculate the final result. This explicit separation and the ability to control each phase make Gatherer a powerful tool for complex, stateful, or stateless aggregations—especially when performance through parallelisation is crucial or custom aggregation logic is required.\nAn example implementation # Let\u0026rsquo;s first define the gatherer in general and put it in a stream.\nGatherer\u0026lt;Integer, ?, ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;\u0026gt; concurrentGatherer = Gatherer.of( initializer, integrator, combiner, finisher ); IntStream.rangeClosed(0, END_INCLUSIVE) .boxed() .parallel() .gather(concurrentGatherer) .forEach(e -\u0026gt; System.out.println(\u0026#34;e.size() = \u0026#34; + e.size())); Now we define the respective subcomponents:\nInitialiser: # var initializer = (Supplier \u0026lt;ConcurrentMap\u0026lt;Integer, List\u0026raquo;) ConcurrentHashMap::new;\nIntegrator: # var integrator = new Gatherer.Integrator\u0026lt; ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;, Integer, ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;\u0026gt;() { @Override public boolean integrate(ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt; state, Integer element, Gatherer.Downstream\u0026lt; ? super ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;\u0026gt; downstream) { if (element \u0026gt; END_INCLUSIVE) return false; //processing can be interrupted int blockStart = (element / 100) * 100; state .computeIfAbsent(blockStart, k -\u0026gt; Collections.synchronizedList(new ArrayList\u0026lt;\u0026gt;())) .add(element); return true; } }; The integrator is responsible for processing individual stream elements (here: Integer) and their insertion into a shared state (ConcurrentMap). The element is sorted according to a specific grouping criterion: all elements that are in the same block of 100 (e.g. 0–99, 100–199, \u0026hellip;), are entered in the same list.\nThere is a special feature in this implementation:\nif (element \u0026gt; END_INCLUSIVE) return false;\nThis condition serves as Abort signal : as soon as an element is above a specified limit (END_INCLUSIVE), processing is completed by returning false canceled. This is a special feature of the Gatherer-Model: The return value false signals that no further elements should be processed - a type early termination.\nstate .computeIfAbsent(blockStart, k -\u0026gt; Collections.synchronizedList(new ArrayList\u0026lt;\u0026gt;())) .add(element); This line does the actual grouping: If under the key blockStart, if no list already exists, a new, synchronised one will be created. ArrayList created.\nThe current item is then added to this list.\nBy using Collections.synchronizedList(\u0026hellip;) becomes even within a parallel gatherer context ensures that list accesses are thread-safe - even though the ConcurrentMap itself is only responsible for map access, not for the values ​​it contains.\nThe integrator therefore defines the following processing semantics:\nElements are grouped by blocks of 100 (0–99, 100–199, etc.). The assignment is done via a ConcurrentMap, where each block contains a list. The lists themselves are synchronised to allow concurrency within the list operations. By returning false can processing ended early become – e.g. B. when a limit value is reached. Combiner: # var combiner = new BinaryOperator\u0026lt;ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;\u0026gt;() { @Override public ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt; apply(ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt; state1, ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt; state2) { var mergedMap = new ConcurrentHashMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;(); // fill in state1 state1.forEach((key, value) -\u0026gt; mergedMap.merge(key, value, (v1, v2) -\u0026gt; { v1.addAll(v2); return v1; }) ); // fill in state 2 state2.forEach((key, value) -\u0026gt; mergedMap.merge(key, value, (v1, v2) -\u0026gt; { v1.addAll(v2); return v1; }) ); return mergedMap; } First, a new empty, thread-safe map is prepared to contain all entries state1 and state2 should be inserted. This new map is deliberately fresh because neither state1 still state2 should be changed - this protects against unwanted side effects and makes the function work referentially transparent.\nvar mergedMap = new ConcurrentHashMap \u0026lt;Integer, List\u0026gt;();\nentries state1 insert # state1 .forEach((k,v) -\u0026gt; { mergedMap .computeIfAbsent(k, k1 -\u0026gt; new ArrayList\u0026lt;\u0026gt;()) .addAll(v); }); This method computeIfAbsent checks whether in the target map mergedMap already an entry for the key k exists. If this is not the case, the lambda is used k1 -\u0026gt; new ArrayList\u0026lt;\u0026gt;() a new entry is created and inserted. The method guarantees that an existing, modifiable list is returned afterwards - regardless of whether it was just created or already existed.\nThe method addAll(\u0026hellip;) hangs all elements of the list v out of state1 to the corresponding list in mergedMap to. This expands the aggregate state for this key in the target map.\nentries state2 insert # The same process is then repeated for state2 repeated:\nstate2.forEach((key, value) -\u0026gt; mergedMap.merge(key, value, (v1, v2) -\u0026gt; { v1.addAll(v2); return v1; }) ); Every entry is made here state2 in the mergedMap transmitted. If the key does not yet exist, the value (value, one List) taken directly. If the key is already in mergedMap, it exists by merge(\u0026hellip;) using a custom merge strategy: the list contents of both maps are merged v1.addAll(v2) combined. What is important here is that v1 is the existing list, and v2 is the newly added one.\nIn the end, the newly created, combined map is returned—it represents the complete aggregate state, which contains the contents of both partial states.\nFinisher: # var finisher = new BiConsumer\u0026lt; ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;, Gatherer.Downstream\u0026lt;? super ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;\u0026gt;\u0026gt;() { @Override public void accept( ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt; state, Gatherer.Downstream\u0026lt;? super ConcurrentMap\u0026lt;Integer, List\u0026lt;Integer\u0026gt;\u0026gt;\u0026gt; downstream) { downstream.push(state); } }; This implementation of the finisher is minimalistic but functionally correct: it takes the final state (state) of the accumulator - one ConcurrentMap\u0026lt;Integer, List\u0026gt; – and hands it directly to him downstream , i.e. the next stage of the processing chain in the stream.\nThe attribute state is the aggregated result of the previous steps (initialiser, integrator, combiner). In this case it is a map that maps blocks of 100 (Integers) to a list of the associated values ​​(List).\nThe attribute downstream is a push receiver that consumes the end result. It abstracts the next processing stage, such as a downstream map, flatMap, or terminal collection process.\nThe method push(\u0026hellip;) of the downstream object explicitly forwards the finished result to the next processing stage. This is fundamentally different from conventional collector Concepts, where the end result is simply returned.\nThis type of handover makes it possible, in particular, in one within Gatherer defined, stateful context to deliver multiple or even conditional results – for example:\nStreaming of intermediate results (e.g. after a specific batch) Quitting early after the first valid result Multiple edition during partitioning In this specific case, however, precisely one result was passed on—the fully aggregated map. This classic “push-forward finisher” determines the condition as a result emitted.\nWe have now examined the Gatherer in detail and pointed out the differences in sequential and parallel processing. So, everything should be together for your first experiments.\nHappy Coding\nSven\n","date":"2 April 2025","externalUrl":null,"permalink":"/posts/rethinking-java-streams-gatherer-for-more-control-and-parallelism/","section":"Posts","summary":"Since version 8, Java has introduced an elegant, functional approach to processing data sets with the Streams API. The terminal operation collect(…) represents the bridge from the stream to a targeted aggregation - in the form of lists, maps, strings or more complex data structures. Until Java 20 the processing was done Collector-Instances were regulated, which internally consisted of a supplier, an accumulator, a combiner and optionally a finisher. This model works well for simple accumulations but has noticeable limitations, particularly for complex, stateful, or conditional aggregations.\n","title":"Rethinking Java Streams: Gatherer for more control and parallelism","type":"posts"},{"content":"The introduction of the Stream API in Java marked a crucial step in the development of functional programming paradigms within the language. With Java 24, stream processing has been further consolidated and is now a central tool for declarative data processing Stream not about an alternative form of Collection, but rather an abstract concept that describes a potentially infinite sequence of data that is transformed and consumed through a pipeline of operations. While a Collection is a data structure that stores data, Stream is a carrier of a computing model: It does not store any data but instead allows the description of data flows.\nGetting started with streams in Java The power of composition Effects on the Classic Design Patterns in Java An example from classic Java to the use of streams Example in classic Java without streams Example in Java using Streams API Changes to the Streams API from Java 8 to 21 - an overview A concrete example of flatMap and mapMulti Conceptual difference Example with flatMap What\u0026rsquo;s new with Java 24 - Gatherer Practical examples with the Gatherer windowFixed windowSliding possible optimizations Each stream processing can be divided into three logically separate phases. The starting point is always a source - one Collection, an array, an I/O channel or a created structure like Stream.generate(\u0026hellip;) be. The source is followed by a number of Intermediate Operations , which transform, filter or sort the Stream. These operations are lazy , d. h. they are not executed immediately but are merely registered. Only through a final Terminal Operation , such as forEach, collect or reduce, execution is triggered and the entire pipeline is concretized. The Laziness principle allows the JVM to optimize, merge, or even eliminate operations without impacting the final result.\nA fundamental feature of stream architecture is its uniqueness. A stream can be consumed exactly once. Once a terminal operation has been performed, the Stream is exhausted. This has profound implications for the design of stream-based algorithms, as one must carefully decide when and where to evaluate a pipeline. Compared to reusable data containers like lists, this requires a shift towards fluent, targeted processing.\nThe structure of a stream pipeline follows a well-defined structure. It begins with the definition of the data source, is completed by a sequence of intermediate operations and ends with a final consumption action. Under the hood, the JVM detects and analyzes this structure and performs a variety of optimizations. For example, the JVM can do operations like map and filter combine (Operation Fusion), activate parallel execution on suitable sources or avoid redundant calculations. These optimization options are only possible due to the declarative nature of streams and underline the paradigmatic difference to imperative code with classic loops.\nIn Java, the Stream API remains a key element for modern, expressive, and concurrent computing. However, understanding them requires a deep awareness of the differences from the conventional collection hierarchy - particularly in terms of laziness, single use, and JVM-powered optimizations that make streams a powerful tool in the Java developer\u0026rsquo;s repertoire.\nGetting started with streams in Java # Getting started with the Stream API in Java requires a basic understanding of functional programming concepts and the way Java integrates these concepts within an object-oriented language. Streams make it possible to describe data flows declaratively, with the focus not on the How but on the Where the data processing lies. This shift from imperative to declarative thinking forms the foundation for a modern, concise and, at the same time, expressive programming model.\nA typical entry point into stream processing is the conversion of existing data structures, such as List- or Setinstances, into a stream. This is done using the method stream(), by most Collection implementations is provided. Alternatively, methods such as: Stream.of(\u0026hellip;), Arrays.stream(\u0026hellip;) or IntStream.range(\u0026hellip;) can be used to create stream instances. Regardless of the source, this always creates a pipeline that is potentially evaluated with a delay.\nAfter a stream is initialized, intermediate operations such as filter, map, or sorted are applied to transform or refine the data flow. These operations are purely descriptive and do not modify the original data source or trigger any calculation. Instead, you build a kind of recipe that is later executed through a terminal operation.\nAs mentioned before, the pipeline is completed by a terminal operation that triggers execution and delivers the result material. Only at this point is the pipeline evaluated, with all previously defined steps being applied to the data in one pass. This is a key difference from traditional processing using loops or iterators, where each processing step is executed immediately.\nEspecially at the beginning, it is helpful to experiment with simple examples, such as filtering character strings or transforming integers. This allows an intuitive understanding of the data flow, the chaining of operations and the underlying execution logic. The advantage of streams quickly becomes apparent: they promote a concise, readable and at the same time efficient expression for data-driven operations, without unnecessarily complicating the structure of the underlying code.\nThe power of composition # The Java Stream API\u0026rsquo;s strength lies in its ability to process data declaratively and, above all, in its ability to form complex processing steps into a compositional unit. Streams allow transformation and filter logic to be defined modularly and combined elegantly, making even complex data flows comprehensible and reusable. The underlying idea is to understand arithmetic operations as functional units that can be fluidly combined into a processing chain.\nFrom a software architecture perspective, stream composition opens an elegant way to modularize processing steps. For example, methods can be defined as those that encapsulate a specific stream transformation and can be embedded as a building block in a more extensive pipeline. By using higher-order functions, for example, by returning a Function\u0026lt;T, R\u0026gt;-Object, a repertoire of reusable processing modules is created that can be put together dynamically. This not only promotes readability, but also testability and maintainability of the code.\nAnother advantage is the clear separation of structure and behaviour. While imperative code typically mixes loop logic with specialized logic, stream composition allows a decoupled view: the type of data flow (e.g. sequential or parallel) is orthogonal to the question of what happens to the data in terms of content. This separation makes it easier to reuse or specifically expand existing pipelines in different contexts without fundamentally changing their structure.\nLast but not least, the combination of different stream types expresses this compositional ability. So primitive streams (IntStream, LongStream, DoubleStream) with object streams via conversion methods like boxed() or mapToInt() be connected. The same can be said about flatMap, which realizes the processing of nested data structures, whereby a stream of containers is created into a flat data stream that can be processed further seamlessly.\nTherefore, the ability to compose streams is more than just a syntactic convenience. It represents a fundamental paradigm shift in the way data flows are designed, structured, and executed in Java. Anyone who masters this approach can convert even complex application logic into clearly structured, functional modules—a gain in terms of both maintainability and expressiveness.\nEffects on the Classic Design Patterns in Java # The integration of the Stream API into Java has profound implications for the use and necessity of classic design patterns. Many patterns that emerged in object-oriented development emerged in response to the lack of functional means of expression. However, with the emergence of streams and their close integration with functional concepts such as lambdas and higher-order functions, the role of these patterns is fundamentally changing. This is particularly noticeable with patterns that were initially used to control the iteration, transformation or filtering of data structures.\nA prominent example of this is the Iterator Pattern , which has been primarily made obsolete by streams. While the Iterator Pattern in classic Java was considered an idiomatic approach to processing collections sequentially, the Stream API completely encapsulates this responsibility. Access to the elements is implicit and declarative; the stream configuration controls the order and traversal behaviour. Manual control over iteration, as provided by the iterator pattern, is thus replaced by a more expressive and, at the same time, less error-prone abstraction.\nThat too Strategy Pattern undergoes a transformation through streams. Strategies that were previously coded as separate classes or interfaces with concrete implementations can now be elegantly implemented via lambdas and method-based composition within stream pipelines. Filtering or mapping strategies that were previously modelled through object-oriented inheritance hierarchies can now be defined inline and dynamically combined - a change that not only simplifies the source code but also increases its flexibility.\nThe Decorator Pattern , traditionally used to extend functionality through nested object structures dynamically, finds a modern equivalent in the stream world in the chaining of intermediate operations. Each operation transforms the Stream and adds another processing layer - not through inheritance or object composition, but through functional pipeline elements. The resulting structure is more compact and allows for a much more dynamic configuration at runtime.\nFurthermore, stream architecture sheds new light on the Template Method Pattern. While this was initially used to define the structure of an algorithm and implement varying steps in subclasses, the Stream API allows such \u0026ldquo;algorithms\u0026rdquo; to be defined via methodically combined functions. The sequence of stream operations specifies the fixed processing steps, while passed functions specify concrete steps. This makes the behaviour more modular and decoupled from static inheritance.\nFinally, streams also change the understanding of patterns like Chain of Responsibility or Pipeline. These patterns are designed to model flexible processing sequences in which each element acts optionally and can pass responsibility. Stream pipelines implement this principle at a functional level, with each intermediate operation corresponding to a \u0026ldquo;processing unit\u0026rdquo;. The big difference is that there is no need to chain objects together manually; instead, the processing chain results from the fluent syntax of the API itself.\nOverall, streams in Java lead to a functional re-contextualization of classic design patterns. This does not make many patterns superfluous but instead transforms them in their implementation. They now appear less rigid class structures and more as dynamic, composable units that seamlessly fit into fluent APIs. For the informed developer, this not only opens up new possibilities for expression but also a reflective understanding of when a pattern in the classic sense is still necessary. But I will report on this in a separate post.\nAn example from classic Java to the use of streams # A classic data processing algorithm in Java extracts, transforms, and aggregates information from a list of complex objects. Let\u0026rsquo;s take a list of as an example Person-Objects, each containing name, age and place of residence. The goal is to extract all the names of adults, sort them alphabetically and return them as a string separated by commas. The classic imperative-object-oriented implementation of this requirement in Java typically takes place via explicit loops, temporary lists and manual control structures.\nYou usually start by creating a new results list in such an implementation. The original data is then iterated over, in one if-Condition it is checked whether the age of the person in question is over 17. If so, the person\u0026rsquo;s name is added to the results list. Once the iteration is complete, the list is sorted alphabetically by explicitly calling the sort method. Finally, the list is created using a loop or by using a StringBuilder converted into a comma-separated string. This approach works correctly but requires a lot of intermediate steps and quickly leads to redundant or error-prone logic - for example, when sorting or formatting the result.\nWith the introduction of the Stream API, the same algorithm can be expressed in a much more concise and declarative manner. The entire data flow – from filtering to transformation to aggregation – can be modeled in a single expression unit. The Stream is derived from the list, then filters filter-Operation to remove adults. A subsequent one map-Transformation extracts the names. The sorting is done by sorted realized, and the final aggregation takes place via Collectors.joining(\u0026quot;,\u0026quot;). The entire algorithm can, therefore, be expressed in a fluent, clearly structured pipeline, the sub-steps of which are semantically self-explanatory through their method names.\nThe differences between the two approaches are both syntactic and conceptual. While the classic solution relies heavily on imperative thinking and controls each processing element manually, the stream approach allows declarative modeling of the \u0026ldquo;what\u0026rdquo; and leaves the \u0026ldquo;how\u0026rdquo; to be executed by the runtime environment. This level of abstraction promotes readability and maintainability of the code, as the developer can concentrate on the technical logic without being distracted by control flow constructs.\nAt the same time, it can be seen that streams also enable semantic compression. An algorithm that previously required several dozen lines is often reduced to a few clearly structured function calls. However, this compactness does not come at the expense of transparency - on the contrary, the descriptive method names and the fluent structure make the data flow explicitly understandable. The stream variant, therefore, not only appears more modern but also conceptually closer to the problem.\nIt can be said that the stream-based implementation integrates functional principles into the Java world without sacrificing type safety or object orientation. It makes it possible to formulate classic algorithms with a new level of abstraction that not only simplifies the code, but also expresses its intent more clearly.\nBut let\u0026rsquo;s look at it in the source code: Let\u0026rsquo;s assume that we have the following structures available for both implementations.\nrecord Person(String name, int age, String city) {} private static final List\u0026lt;Person\u0026gt; PEOPLE = List.of( new Person(\u0026#34;Alice\u0026#34;, 23, \u0026#34;Berlin\u0026#34;), new Person(\u0026#34;Bob\u0026#34;, 16, \u0026#34;Hamburg\u0026#34;), new Person(\u0026#34;Clara\u0026#34;, 19, \u0026#34;München\u0026#34;), new Person(\u0026#34;David\u0026#34;, 17, \u0026#34;Köln\u0026#34;) ); Example in classic Java without streams # List\u0026lt;String\u0026gt; adultNames = new ArrayList\u0026lt;\u0026gt;(); for (Person p : PEOPLE) { if (p.age \u0026gt;= 18) { adultNames.add(p.name); } } Collections.sort(adultNames); StringBuilder result = new StringBuilder(); for (int i = 0; i \u0026lt; adultNames.size(); i++) { result.append(adultNames.get(i)); if (i \u0026lt; adultNames.size() - 1) { result.append(\u0026#34;, \u0026#34;); } } System.out.println(\u0026#34;Ergebnis: \u0026#34; + result); } In the classic variant, the data flow is fragmented and distributed over several steps: filtering, collecting, sorting and formatting take place in separate sections, sometimes using temporary structures. This leads to increased complexity and potential sources of errors - for example when formatting the output string correctly.\nExample in Java using Streams API # public static void main(String[] args) { String result = PEOPLE.stream() .filter(p -\u0026gt; p.age() \u0026gt;= 18) .map(Person::name) .sorted() .collect(Collectors.joining(\u0026#34;, \u0026#34;)); System.out.println(\u0026#34;Ergebnis: \u0026#34; + result); } The stream variant, on the other hand, expresses the entire data processing process in a single pipeline. Each processing step is clearly named, the transformation takes place fluidly, and the result is immediately visible. Particularly noteworthy is the better one readability , the Reduction of side effects and the Extensibility – additional processing steps can be added by simply inserting additional operations into the pipeline without fundamentally changing the structure.\nChanges to the Streams API from Java 8 to 21 - an overview # Since its introduction in Java 8, the Stream API has fundamentally changed the way data is processed in Java. With the aim of enabling declarative, functionally inspired data flows, she established a new programming paradigm within the object-oriented language. The initial version was already remarkably expressive: it offered a clear separation between data source, transformation and terminal operation, supported lazy evaluation and could be executed both sequentially and in parallel. In particular the integration with lambdas, method references and the java.util.functionlibrary enabled a fluid, type-safe style of data processing that was previously only possible via external libraries or explicit iterators.\nWith the subsequent versions of the JDK, the Stream API was not fundamentally redesigned, but was continually expanded, refined and stabilized. Java 9 brought with it the first significant enhancements, such as the methods causeWhile, dropWhile and iterate with predicates that made it possible to control stream pipelines even more precisely and terminate them early. These extensions closed a semantic gap in the original API and, in particular, improved the ability to model more complex data flows. Also the introduction of Optional. Stream () was a notable move as it provided an elegant bridge between the types Optional and Stream and thus further increased the ability to compose in a functional style.\nIn Java 10 to 14, priority was given to accompanying language features such as our introduced, which did not directly change the Stream API itself, but improved its readability and applicability in many contexts. It was only with Java 16 and the consistent spread of records as compact data containers that stream processing became more expressive again, as it could now be combined with structured but immutable data models - a clear advantage for parallel or deterministic processing scenarios.\nJava 17, as a long-term support version, consolidated the API through additional optimizations in the backend and further improved integration with pattern matching and modern sealed-Class hierarchy. These structural improvements did not introduce new methods in the Stream class itself, but enabled more precise handling of heterogeneous data flows and the formulation of domain-specific pipelines at a high level of abstraction.\nIt was only Java 21 that brought tangible expansions in the area of ​​functional data processing, especially in interaction Scoped Values, structured concurrency and the further developed ForkJoinPool implementation. While the Stream API received hardly any new methods formally, its applicability in the context of concurrent and memory-optimized architectures was significantly strengthened. The increased performance with parallel execution and the more efficient management of intermediate results also reflect a maturation of the API at the implementation level. With Java 21, streams can not only be written more elegantly but also executed more securely and predictably - especially in asynchronous or reactive processing environments.\nIn summary, the Stream API has undergone a clear maturation process from Java 8 to Java 21. While its conceptual core remained essentially constant, its semantic scope was clarified, its integration with modern linguistic means was deepened, and its efficiency was optimized on several levels.\nA concrete example of flatMap and mapMulti # Conceptual difference # flatMap was already introduced with Java 8 and is based on the idea of ​​applying a function that creates a new one for each element of the original stream Stream produced. These nested streams are then \u0026ldquo;flat\u0026rdquo; merged so that the resulting Stream has a flat structure. This pattern is compelling and allows elegant transformations of nested data structures - for example Stream\u0026lt;List\u0026gt; to Stream .\nmapMulti(), introduced in Java 16, takes a different approach. Instead of creating nested streams, one is created here Consumer based Interface used: The method receives one for each element of the Stream BiConsumer, through which the developer can directly deliver zero, one or more output values ​​to the downstream - without creating temporary data structures such as lists or streams. This avoids heap allocations, which plays a particularly relevant role in performance-critical scenarios.\nIn the following example, let\u0026rsquo;s assume that we have the following data structure.\nprivate static List\u0026lt;List\u0026lt;String\u0026gt;\u0026gt; DATA = List.of( List.of(\u0026#34;a\u0026#34;, \u0026#34;b\u0026#34;), List.of(\u0026#34;c\u0026#34;), List.of(), List.of(\u0026#34;d\u0026#34;, \u0026#34;e\u0026#34;) ); Example with flatMap # List\u0026lt;String\u0026gt; result = DATA .stream() .flatMap(List::stream) .collect(Collectors.toList()); Here each inner list is converted into a stream and then \u0026ldquo;flattened\u0026rdquo;.\nList\u0026lt;String\u0026gt; result = DATA .stream() .\u0026lt;String\u0026gt;mapMulti(Iterable::forEach) .toList(); In this case, no new stream instance is created. Instead, it is about the forEach-Loop the content passed directly to the consumer.\nflatMap stands more for declarative clarity and conceptual simplicity. The method represents a fundamental principle of functional programming and is therefore particularly suitable if the transformation is based on existing Stream-generating functions takes place. It\u0026rsquo;s great for teaching, readability, and functional reasoning, but does introduce some overhead from temporary streams.\nmapMulti(), however, aims to optimize away. It allows transformations to be expressed more efficiently by granting direct control over the output. This reduces heap allocations, allows better memory locality and is therefore preferable from a JVM optimization perspective if performance is a priority. However, the semantic expression is more complex: the user must explicitly say so Consumer-Logic work, which can limit readability and comprehensibility for functionally less experienced developers.\nYou can say that flatMap, the idiomatic, functional way, remains, while mapMulti() represents a targeted tool for high-performance data processing. Their coexistence within the Stream API demonstrates Java\u0026rsquo;s ambition to combine both expressiveness and efficiency within the same paradigm - a balancing act between declarative elegance and system-level control.\nWhat\u0026rsquo;s new with Java 24 - Gatherer # With Java 24, the Stream API has been expanded to include a new concept called Gatherer company. This addition addresses a long-standing gap in stream processing: the ability to perform custom aggregations across multiple elements in a controlled, stateful manner - but not at the end of the pipeline, as is the case with Collector-Instances is the case, but while of the stream process itself, as an integral part of the transformation. Gatherers represent a new class of intermediate operations that were specifically designed for advanced data flow logic.\nThe fundamental motivation for introducing gatherers is the limitation of the previous API in dealing with compound, multi-step, or stateful operations, particularly in the case of transformations that do not just produce a single element per input, but require sequencing, delaying, or grouping across multiple input elements. Previous tools like map, flatMap or peek all operate either statelessly or with limited context. For more complex cases, specialized iterator implementations or external libraries had to be used - which contradicts the goal of declarative data flow modelling in streams.\nGatherers solve this problem through a new processing model based on a controlled state. Similar to CollectorInstances define a set of functions that allow elements to be buffered, transformed and emitted - not at the end of the Stream, but as part of the ongoing data flow. So they combine the conceptual strengths of Collector and mapMulti, but extend these to include explicit support for temporary states and flexible emission strategies. There will be one Sink used to deliver any results downstream - even several per input element or even delayed.\nThe purpose of this expansion lies not only in expressivity but also in performance. Many use cases, such as sliding windows, temporal aggregations, sequence analysis or transformation logic with history, can be modelled precisely and efficiently with gatherers without sacrificing the declarative structure of the pipeline. This provides the developer with a tool that explicitly allows intermediate states and context dependency within stream processing but is type-safe, controlled and idiomatically embedded in the existing API.\nThis opens up new perspectives in development. Gatherers enable complex data flow modelling with a declarative character without falling back into imperative control logic. They significantly expand the semantic space of streams and thus represent a logical next step in the evolution of stream architecture - comparable to the introduction of Collector or mapMulti. Its introduction shows that Java integrates functional principles and develops them further in state models, process control and efficiency - without the compromises that often accompany system-level optimization.\nPractical examples with the Gatherer # The introduction of gatherers not only created a new extension of stream processing but also provided a collection of predefined gatherers that address typical use cases that were previously difficult to model. These predefined gatherers combine declarative expressiveness with stateful transformation, enabling new forms of data stream processing - especially for sequential, grouped or sequentially correlated data flows. The central representatives of this new genre include windowFixed, windowSliding, fold and scan. Each of these gatherers brings specific behaviour that previously had to be implemented with considerable manual effort or even outside of the stream API.\nwindowFixed # The Gatherer windowFixed allows a stream to be divided into equally sized, non-overlapping windows. This is particularly relevant if data is to be further processed or aggregated in blocks. Suppose we want to split a list of integers into groups of five and calculate the sum of each group. With windowFixed This is very elegant:\nList\u0026lt;Integer\u0026gt; input = IntStream.rangeClosed(1, 15).boxed().toList(); List\u0026lt;Integer\u0026gt; resultGatherer = input.stream() .gather(Gatherers.windowFixed(5)) .map(window -\u0026gt; window.stream().reduce(0, Integer::sum)) .toList(); // [15, 40, 65] System.out.println(\u0026#34;resultGatherer = \u0026#34; + resultGatherer); The alternative without a gatherer could look like this.\nList\u0026lt;Integer\u0026gt; result = new ArrayList\u0026lt;\u0026gt;(); List\u0026lt;Integer\u0026gt; window = new ArrayList\u0026lt;\u0026gt;(); for (int i : input) { window.add(i); if (window.size() == 5) { result.add(window.stream().reduce(0, Integer::sum)); window.clear(); } } System.out.println(\u0026#34;result = \u0026#34; + result); The difference is clear: The Gatherer variant is not only more compact, but also clearly separates data flow and logic, while the imperative solution works with side effects and state management.\nwindowSliding # A related but semantically more sophisticated case is given by windowSliding covered. These are sliding windows - each new element moves the window by one.\nList\u0026lt;Integer\u0026gt; input = List.of(1, 2, 3, 4, 5); List\u0026lt;Double\u0026gt; result = input.stream() .gather(Gatherers.windowSliding(3)) .map(window -\u0026gt; window.stream().mapToInt(Integer::intValue).average().orElse(0)) .toList(); // [2.0, 3.0, 4.0] The given Java source code demonstrates an application des Gatherers windowSliding(n). This is used in this example to create sliding windows - so-called - over a given list of integers Sliding Windows – to generate. The database in this case is an immutable list of integers:\nList input = List.of(1, 2, 3, 4, 5);\nThe subsequent stream processing aims to calculate the arithmetic mean (average) of the elements contained over each sliding window of size three. The method windowSliding(3) causes the data stream to be internally split into overlapping windows, with each window containing three consecutive elements. For the list [1, 2, 3, 4, 5] This results in the following windows:\n[1, 2, 3] [2, 3, 4] [3, 4, 5] Within the mapoperator will turn each of these windows into one Stream converted, where with mapToInt(Integer::intValue).average() the average value is calculated. This method provides one OptionalDouble, from which means orElse(0) a double is extracted if the window is empty (which is impossible in this use case because the window size is smaller or equal to the initial list).\nThe resulting List contains the average values ​​of the overlapping triples and looks specifically as follows:\n[2.0, 3.0, 4.0]\nThe code\u0026rsquo;s declarative style promotes readability and understanding, while the Gatherers-API provides an elegant way to perform sliding aggregations directly in the stream context without resorting to manual window logic or imperative loops.\npossible optimizations # Is Several optimization goals can be identified that can improve both the robustness against runtime errors and the performance and semantic readability of the code. Each of these goals brings different requirements for the design of the code.\nRegarding performanceoptimization , it can be determined that the current source code creates unnecessary objects by repeatedly creating temporary ones, such as stream objects within the mapping. This can lead to a noticeable increase in the garbage collection load, especially with large amounts of data. To increase performance, you should ensure repeated use of Stream () within the window and instead directly with the existing one List work provided through windowSliding. An alternative solution could look like this:\nList\u0026lt;Double\u0026gt; result = input.stream() .gather(Gatherers.windowSliding(3)) .map(window -\u0026gt; { int sum = 0; for (int i : window) sum += i; return sum / 3.0; }) .toList(); This version completely eliminates the overhead of internal stream processing per window. Instead, the summation is done directly via a simple loop. This not only reduces the allocation of temporary objects, but allows the JVM to perform aggressive inlining and loop unrolling optimizations. In addition, the division by the window size is constant, since windowSliding(3) a fixed window size guaranteed. However, if the window size varies dynamically, you could alternatively window.size() determine at runtime without endangering semantic correctness.\nRegarding the readability , It should be noted that functional elegance and imperatively expressed clarity are not necessarily in contradiction. The original version with mapToInt(\u0026hellip;).average().orElse(0) Although it is syntactically compact, it is less immediately understandable for many developers, especially with regard to the treatment of OptionalDouble. The imperative variant with direct summation and division, on the other hand, clearly reveals the underlying intention - the calculation of an average over three elements - without a deep understanding of the Stream-API is required. For teams with heterogeneous levels of experience or in code bases with a strong focus on maintainability, the following version is often preferable:\nList\u0026lt;Double\u0026gt; result = input.stream() .gather(Gatherers.windowSliding(3)) .map(window -\u0026gt; { int sum = 0; for (Integer value : window) { sum += value; } return (double) sum / window.size(); }) .toList(); This variant avoids potentially confusing method chains and expresses the calculation step by step. This not only makes the code easier to understand, but also easier to test, as each calculation step can be validated individually. At the same time, it remains compact and uses modern Java constructs such as Stream.gather() in an idiomatic way.\nOverall, it can be seen that by taking robustness, performance and readability into account, both the functional and structural quality of the code can be increased. The choice of specific optimization always depends on the application context and the requirements for runtime behavior, error handling and maintainability.\nThat should be it at this point for now. In the following post I will go into more detail about the practical use of streams, the use of gatherers and similar things.\nHappy Coding\nSven\n","date":"29 March 2025","externalUrl":null,"permalink":"/posts/from-java-8-to-24-the-evolution-of-the-streams-api/","section":"Posts","summary":"The introduction of the Stream API in Java marked a crucial step in the development of functional programming paradigms within the language. With Java 24, stream processing has been further consolidated and is now a central tool for declarative data processing Stream not about an alternative form of Collection, but rather an abstract concept that describes a potentially infinite sequence of data that is transformed and consumed through a pipeline of operations. While a Collection is a data structure that stores data, Stream is a carrier of a computing model: It does not store any data but instead allows the description of data flows.\n","title":"From Java 8 to 24: The evolution of the Streams API","type":"posts"},{"content":"","date":"23 November 2024","externalUrl":null,"permalink":"/tags/cuda/","section":"Tags","summary":"","title":"CUDA","type":"tags"},{"content":"","date":"23 November 2024","externalUrl":null,"permalink":"/tags/gpu/","section":"Tags","summary":"","title":"GPU","type":"tags"},{"content":"","date":"23 November 2024","externalUrl":null,"permalink":"/tags/opencl/","section":"Tags","summary":"","title":"OpenCL","type":"tags"},{"content":"","date":"23 November 2024","externalUrl":null,"permalink":"/tags/performance/","section":"Tags","summary":"","title":"Performance","type":"tags"},{"content":"","date":"23 November 2024","externalUrl":null,"permalink":"/tags/tornadovm/","section":"Tags","summary":"","title":"TornadoVM","type":"tags"},{"content":"TornadoVM is an open-source framework that extends the Java Virtual Machine (JVM) to support hardware accelerators such as Graphics Processing Units (GPUs), Field-Programmable Gate Arrays (FPGAs), and multi-core central processing units (CPUs). This allows developers to accelerate their Java programs on heterogeneous hardware without needing to rewrite their code in low-level languages such as CUDA or OpenCL.\nMotivation and Background How TornadoVM Works Key Features Use Cases Performance and Benchmarks Limitations and Challenges Integration with Java Ecosystem Future Directions Theoretical Background: Parallelism in Matrix Multiplication How TornadoVM Exploits Parallelism Example: Matrix Multiplication in Java with TornadoVM 1. Code Explanation Theoretical Discussion Performance Considerations Here\u0026rsquo;s an overview of TornadoVM\u0026rsquo;s key aspects and features:\nMotivation and Background # The growth in heterogeneous computing, driven by the increasing demand for performance in fields like machine learning, data processing, and scientific computing, has led to the development of specialised hardware such as GPUs and FPGAs. These devices offer significant performance improvements over traditional CPUs for parallel workloads.\nHowever, programming for these accelerators traditionally requires low-level programming models like CUDA or OpenCL, which are less developer-friendly than higher-level languages like Java. TornadoVM addresses this challenge by allowing developers to use the Java programming language while transparently offloading computations to hardware accelerators.\nThe JVM has been traditionally optimised for multi-core CPUs, but its design is unsuited for exploiting massively parallel architectures like GPUs. TornadoVM fills this gap by introducing a runtime and compilation infrastructure that allows the JVM to leverage these devices.\nHow TornadoVM Works # TornadoVM provides an intermediate layer between Java applications and the underlying hardware. It intercepts certain portions of the Java code, such as loops and parallelizable methods, and compiles them to run on GPUs, FPGAs, or multi-core CPUs.\nThe TornadoVM runtime is divided into three main components:\nBytecode Analyzer : TornadoVM analyses the Java bytecode to identify portions of the code that can be accelerated. This typically includes loops, data-parallel operations, and compute-intensive methods.\nJIT Compiler : TornadoVM uses a Just-In-Time (JIT) compiler to translate the selected portions of the bytecode into OpenCL or PTX (NVIDIA\u0026rsquo;s parallel thread execution format) code that can run on GPUs or FPGAs. The compiler also optimises the code for the specific hardware target.\nTask Scheduling and Execution : Once compiled, TornadoVM schedules the execution tasks on the available hardware. It manages data transfers between the host (CPU) and the accelerators and ensures the results are correctly integrated into the running Java program.\nKey Features # Java API : TornadoVM provides a simple Java API that allows developers to annotate the code they want to accelerate. This can include compute-intensive methods or loops suitable for parallel execution.\nAutomatic Offloading : TornadoVM automatically detects which portions of the code can be offloaded to the hardware accelerators. The developer doesn\u0026rsquo;t need to manage low-level details such as memory transfers or kernel execution.\nMulti-backend Support : TornadoVM supports multiple backends, including OpenCL, PTX (for NVIDIA GPUs), and SPIR-V. This enables it to target various hardware platforms, from different GPU vendors (NVIDIA, AMD, Intel) to FPGAs.\nData Management : TornadoVM handles data transfers between the main memory (used by the CPU) and the memory of the accelerators (such as the GPU memory). It uses a memory management system to minimise unnecessary data transfers, reduce overhead, and improve performance.\nHeterogeneous Task Scheduling : TornadoVM can execute multiple tasks across different devices in parallel. For example, it can execute one part of the code on a CPU and another part on a GPU, enabling efficient use of all available hardware resources.\nUse Cases # TornadoVM is particularly suited for applications that can benefit from parallel execution. Some of the most common use cases include:\nMachine Learning : Many machine learning algorithms, especially those based on matrix operations (like neural networks), can be accelerated on GPUs or FPGAs using TornadoVM.\nBig Data Processing : Frameworks like Apache Spark can be integrated with TornadoVM to accelerate data processing pipelines. By offloading certain operations to GPUs, it can significantly reduce the time required to process large datasets.\nScientific Computing : Many scientific applications involve complex mathematical computations that can benefit from parallel execution on hardware accelerators. TornadoVM makes it possible to run these applications on GPUs without rewriting the code in CUDA or OpenCL.\nFinancial Modeling : Algorithms used in financial simulations, such as Monte Carlo simulations or options pricing, often involve parallelizable computations that can be offloaded to hardware accelerators using TornadoVM.\nPerformance and Benchmarks # TornadoVM has been shown to improve performance significantly for various applications. For example, benchmarks have demonstrated that TornadoVM can speed up certain machine learning algorithms by 10x to 100x when running on GPUs, compared to the same code running on a CPU.\nThe performance gains are particularly significant for compute-bound applications that involve a large number of parallel operations. However, not all applications will benefit from TornadoVM. Applications that are I/O-bound or involve a lot of sequential processing may not see significant speedups.\nLimitations and Challenges # Despite its benefits, TornadoVM has some limitations:\nDevice Availability : TornadoVM requires the presence of a hardware accelerator (such as a GPU or FPGA) to provide performance improvements. If no such device is available, the code will fall back to running on the CPU.\nLimited Scope : TornadoVM is most effective for applications that involve parallelizable computations. Applications with complex control flow or those that are heavily reliant on I/O may not see significant performance improvements.\nHardware-Specific Tuning : While TornadoVM abstracts many of the details of hardware acceleration, developers may still need to fine-tune their code for specific hardware platforms to achieve optimal performance.\nIntegration with Java Ecosystem # One of the key strengths of TornadoVM is its seamless integration with the existing Java ecosystem. It works with the standard Java Development Kit (JDK) and can be integrated with popular Java frameworks such as:\nApache Spark : TornadoVM can be used to accelerate Spark workloads by offloading certain operations to hardware accelerators. This can significantly reduce the time required to process large datasets.\nJVM-based Languages : TornadoVM can be used with any JVM-based language, including Scala, Kotlin, and Groovy. This makes it a versatile solution for developers working in various JVM languages.\nGraalVM : TornadoVM can also be used in conjunction with GraalVM, an advanced JVM that includes an optimising compiler and support for polyglot programming. GraalVM can further enhance TornadoVM\u0026rsquo;s performance by applying additional optimizations at runtime.\nFuture Directions # TornadoVM is an actively developed project, and its developers are working on several features to improve its functionality and performance. Some of the key areas of future development include:\nExpanded Hardware Support : TornadoVM continually supports new hardware platforms, including newer generations of GPUs and FPGAs.\nImproved Optimization : The TornadoVM team is working on improving the JIT compiler to generate more optimised code for hardware accelerators, leading to even more significant performance gains.\nAutomatic Parallelization : While TornadoVM already supports automatic offloading of specific code segments, there is ongoing work to automate further the process of identifying parallelizable code and offloading it to hardware accelerators.\nDeep Integration with Machine Learning Frameworks : TornadoVM is exploring deeper integration with machine learning frameworks like TensorFlow and PyTorch to provide hardware acceleration for a wider range of machine learning tasks.\nTornadoVM is a powerful tool that brings hardware acceleration to the Java ecosystem, allowing developers to harness the power of GPUs, FPGAs, and multi-core CPUs without having to leave the familiar world of the JVM. By simplifying the process of offloading computations to heterogeneous hardware, TornadoVM opens up new possibilities for accelerating a wide range of applications, from machine learning and big data processing to scientific computing and financial modelling. As the demand for high-performance computing continues to grow, TornadoVM is likely to play an increasingly important role in enabling Java developers to take full advantage of modern hardware platforms.\nLet\u0026rsquo;s have an example # To give you a comprehensive example in Java using TornadoVM, we will look at simple matrix multiplication, a common parallelisable operation. The example will include a theoretical background on parallelism, how TornadoVM exploits it, and how the program is structured to benefit from GPU/FPGA acceleration.\nTheoretical Background: Parallelism in Matrix Multiplication # Matrix multiplication is a good example of parallel execution because each element of the result matrix can be computed independently of the others. Consider two matrices, A and B, where:\n- Matrix A is of size N x M,\n- Matrix B is of size M x K.\nThe resulting matrix C will be of size N x K, where each element C[i][j] is the dot product of the i-th row of A and the j-th column of B:\nThis operation involves a large number of independent computations, meaning that all elements of C can be computed in parallel. This is an ideal scenario for hardware acceleration using GPUs or FPGAs, which can perform a massive number of operations simultaneously.\nHow TornadoVM Exploits Parallelism # TornadoVM can accelerate the matrix multiplication by identifying the parallelisable loops (i.e., loops where the computations are independent). It translates the loops into tasks that can be distributed across GPU cores. The key steps are:\nBytecode Analysis : TornadoVM inspects the Java bytecode to identify parallelisable sections, such as loops where each iteration performs independent operations.\nJIT Compilation : The identified sections are compiled Just-In-Time (JIT) into code that runs on the accelerator (such as PTX for NVIDIA GPUs).\nTask Execution : The compiled tasks are executed on the available hardware, while data is transferred between the host (CPU) and the accelerator as needed.\nExample: Matrix Multiplication in Java with TornadoVM # Below is a Java code example that uses TornadoVM to perform matrix multiplication.\nimport uk.ac.manchester.tornado.api.TaskSchedule; import uk.ac.manchester.tornado.api.TornadoVM; import uk.ac.manchester.tornado.api.annotations.Parallel; public class MatrixMultiplication { // Matrix multiplication using TornadoVM public static void multiplyMatrix(float[][] A, float[][] B, float[][] C, int N, int M, int K) { // TornadoVM annotation to indicate parallelism @Parallel for (int i = 0; i \u0026lt; N; i++) { for (int j = 0; j \u0026lt; K; j++) { float sum = 0.0f; for (int k = 0; k \u0026lt; M; k++) { sum += A[i][k] * B[k][j]; } C[i][j] = sum; } } } public static void main(String[] args) { int N = 1024; // Number of rows in A int M = 1024; // Number of columns in A (and rows in B) int K = 1024; // Number of columns in B // Initialize matrices float[][] A = new float[N][M]; float[][] B = new float[M][K]; float[][] C = new float[N][K]; // Result matrix // Fill matrices A and B with random values for (int i = 0; i \u0026lt; N; i++) { for (int j = 0; j \u0026lt; M; j++) { A[i][j] = (float) Math.random(); } } for (int i = 0; i \u0026lt; M; i++) { for (int j = 0; j \u0026lt; K; j++) { B[i][j] = (float) Math.random(); } } // Create a task schedule for TornadoVM TaskSchedule ts = new TaskSchedule(\u0026#34;s0\u0026#34;) .task(\u0026#34;t0\u0026#34;, MatrixMultiplication::multiplyMatrix, A, B, C, N, M, K) .mapAllTo(TornadoVM.getDefaultDevice()); // Execute the task on the GPU ts.execute(); // Print out a few results for verification System.out.println(\u0026#34;Result (C[0][0]): \u0026#34; + C[0][0]); System.out.println(\u0026#34;Result (C[10][10]): \u0026#34; + C[10][10]); } } Code Explanation # Matrix Initialization :\nMatrices A and B are initialised with random floating-point values. The matrix C will store the result of the multiplication.\nMatrix Multiplication Function :\nThe multiplyMatrix method multiplies matrix A by matrix B and stores the result in matrix C. The @Parallel annotation tells TornadoVM that this loop is parallelisable, meaning that each computation of C[i][j] can be executed independently across different threads or hardware cores.\nTask Scheduling with TornadoVM :\nThe TaskSchedule object schedules a TornadoVM task, where the multiplyMatrix function is executed. .mapAllTo(TornadoVM.getDefaultDevice()) specifies that the entire task will be mapped to the default TornadoVM device, which could be a GPU, FPGA, or multi-core CPU, depending on the available hardware. The .execute() method runs the task, performing the matrix multiplication on the selected hardware accelerator.\nPerformance Consideration :\nFor large matrix sizes (e.g., 1024x1024, as used in the example), the GPU can perform the multiplication much faster than a CPU by exploiting parallelism. TornadoVM handles the data transfer and task offloading, ensuring efficient execution on the accelerator.\nTheoretical Discussion # Parallelism :\nEach element C[i][j] is independent of all other elements in matrix multiplication, making the operation embarrassingly parallel. The GPU can execute these independent operations concurrently with its thousands of cores, leading to significant speedups over a sequential CPU-based implementation.\nData Transfer :\nTornadoVM manages data transfers between the CPU (host) and the GPU (device). Before the computation starts, matrices A and B are copied to the GPU\u0026rsquo;s memory. After the computation, the resulting matrix C is transferred back to the CPU. TornadoVM optimised these transfers, ensuring that only the necessary data is copied and avoiding redundant transfers to minimise overhead.\nMemory Coalescing :\nMemory access patterns are important for performance on a GPU. TornadoVM attempts to optimise memory access through techniques like memory coalescing, where adjacent threads access adjacent memory locations, reducing the number of memory transactions and improving bandwidth utilisation.\nLoop Parallelization :\nThe loop structure is a perfect candidate for parallelisation: TornadoVM splits the outer two loops (over i' and jindices) across different GPU threads. Each thread computes a single element of the result matrixC, while the innermost loop (over k`) can be executed within each thread.\nJust-in-Time Compilation :\nTornadoVM\u0026rsquo;s JIT compiler generates GPU-specific code (such as PTX code for NVIDIA GPUs) based on the annotated Java bytecode. This process occurs at runtime, allowing TornadoVM to optimise the code for the specific hardware platform in use.\nPerformance Considerations # Using TornadoVM on a GPU or FPGA for matrix multiplication can yield significant performance improvements, especially for large matrices. For instance:\nA CPU may take several seconds to multiply two large matrices (e.g., 1024x1024), whereas a GPU can perform the same operation in a fraction of the time by leveraging its thousands of cores. The larger the matrices, the more efficient TornadoVM becomes, as the overhead of transferring data between the CPU and the accelerator becomes less significant relative to the computation time.\nThis example illustrates how TornadoVM allows Java developers to exploit hardware accelerators like GPUs for computationally expensive tasks like matrix multiplication. By identifying parallelisable loops and offloading them to the accelerator, TornadoVM achieves substantial performance improvements while allowing developers to continue working in Java without needing lower-level languages like CUDA or OpenCL.\nHappy Coding\nSven\n","date":"23 November 2024","externalUrl":null,"permalink":"/posts/tornadovm-boosting-the-concurrency/","section":"Posts","summary":"TornadoVM is an open-source framework that extends the Java Virtual Machine (JVM) to support hardware accelerators such as Graphics Processing Units (GPUs), Field-Programmable Gate Arrays (FPGAs), and multi-core central processing units (CPUs). This allows developers to accelerate their Java programs on heterogeneous hardware without needing to rewrite their code in low-level languages such as CUDA or OpenCL.\n","title":"TornadoVM - Boosting the Concurrency","type":"posts"},{"content":"","date":"13 November 2024","externalUrl":null,"permalink":"/tags/cache/","section":"Tags","summary":"","title":"Cache","type":"tags"},{"content":"Cache poisoning on Maven Caches is a specific attack that targets how Maven Caches manages packages and dependencies in a software development process. It\u0026rsquo;s essential to understand how Maven works before we look at the details of cache poisoning.\nOverview of Maven and its caches What is cache poisoning? Types of cache poisoning on Maven caches Man-in-the-Middle (MITM) Cache Poisoning Exploit repository vulnerabilities Dependency Confusion Basics of Dependency Confusion How Dependency Confusion Was Discovered Why does Dependency Confusion work so effectively? Typosquatting Basics of typosquatting Typical typosquatting techniques Dangers of typosquatting Maven typosquatting cases Process of a cache poisoning attack on Maven Security mechanisms to protect against cache poisoning Regular updates and patch management Using HTTPS Signature verification Improve repository security Dependency Scanning and Monitoring Version Pinning Private Maven-Repositories Implementation of code reviews and security checks Understanding CVEs in the context of Maven and cache poisoning Relevant CVEs related to Maven and repository management tools CVE-2020-13949: Remote Code Execution in Apache Maven CVE-2021-25329: Denial of Service in Maven Artifact Resolver CVE-2019-0221: Directory Traversal in Sonatype Nexus Repository Manager CVE-2022-26134: Arbitrary File Upload in JFrog Artifactory CVE-2021-44228: Log4Shell (Log4j) Analysis and impact of the mentioned CVEs Future challenges and continuous security improvements Conclusion Overview of Maven and its caches # Apache Maven is a widely used build management tool in Java projects. It automates the dependency management, build process, and application deployment. However, some fundamental mechanisms make it necessary to consider the security of repositories and dependencies when using Maven.\nMaven uses repositories to manage libraries and dependencies. There are two types of Maven repositories:\nLocal repository : A copy of all downloaded libraries and dependencies is saved on the local machine.\nRemote Repositories : Maven can access various remote repositories, such as the central Maven repository or a company\u0026rsquo;s custom repositories.\nAfter downloading them from a remote repository, Maven stores all dependencies in the local repository (cache). This allows dependencies that are needed multiple times to load more quickly because they do not have to be downloaded from a remote repository each time.\nWhat is cache poisoning? # Cache poisoning is a class of attacks in which an attacker fills a system\u0026rsquo;s cache (in this case Maven caches) with manipulated or malicious content. This results in legitimate requests that should receive the original data instead of the data injected by an attacker. In Maven terms, cache poisoning refers to when an attacker injects malicious artefacts into a developer\u0026rsquo;s or build\u0026rsquo;s cache by exploiting a vulnerability in the Maven build process or repository servers.\nThe attack aims to deliver malicious dependencies that are then integrated into software projects. These poisoned dependencies could contain malicious code to steal sensitive data, take control of the system, or sabotage the project.\nTypes of cache poisoning on Maven caches # There are several scenarios in which cache poisoning attacks can be carried out on Maven repositories:\nMan-in-the-Middle (MITM) Cache Poisoning # A man-in-the-middle attack allows an attacker to intercept and manipulate network traffic between the developer and the remote Maven repository. If communication is not encrypted, an attacker can inject crafted artefacts and introduce them into the local Maven cache. As a result, the developer believes that the dependencies come from a trusted repository, when in fact, they have been tampered with.\nSuch an attack could be successful if Maven communicates with repositories over unsecured HTTP connections. The central Maven repository (Maven Central) now exclusively uses HTTPS to prevent such attacks, but some private or legacy repositories use HTTP.\nExploit repository vulnerabilities # If an attacker gains access to the remote repository, they can upload arbitrary artefacts or replace existing versions. This happens, for example, if the repository is poorly secured or a vulnerability in the repository management tool (like Nexus or Artifactory) is exploited. In this case, the attacker can inject malware directly into the repository, causing developers worldwide to download the compromised artefact and store it in their Maven cache.\nDependency Confusion # A particularly dangerous attack vector that has received much attention in recent years is the so-called \u0026ldquo;dependency confusion\u0026rdquo; attack. This attack is because many modern software projects draw dependencies from internal and private repositories and public repositories such as Maven Central. The main goal of a Dependency Confusion attack is to inject malicious packages via publicly accessible repositories into a company or project that believes it is using internal or private dependencies.\nBasics of Dependency Confusion # Many companies and projects maintain internal Maven repositories where they store their own libraries and dependencies that are not publicly accessible. These internal libraries can implement specific functionalities or make adaptations to public libraries. Developers often define the name and version of dependencies in the Maven configuration (pom.xml) without realising that Maven prioritises dependencies, favouring public repositories like Maven Central over internal ones unless explicitly configured otherwise.\nA dependency confusion attack exploits exactly this priority order. The attacker publishes a package with the same name as an internal library to the public Maven repository, often with a higher version number than the one used internally. When Maven then looks for that dependency, it usually prefers the publicly available package rather than the private internal version. This downloads the malicious package and stores it in the developer\u0026rsquo;s Maven cache, from where it will be used in future builds.\nHow Dependency Confusion Was Discovered # A security researcher named Alex Birsan popularised this attack in 2021 when he demonstrated how easy it was to poison dependencies in projects at major tech companies. By releasing packages with the same names as internal libraries of large companies such as Apple, Microsoft, and Tesla, he successfully launched dependency confusion attacks against these companies.\nBirsan did not use malicious content in his attacks but harmless code to prove that the system was vulnerable. He was able to show that in many cases, the companies\u0026rsquo; build systems had downloaded and used the malicious (in his case harmless) package instead of the real internal library. This disclosure led to massive awareness in the security community about the risks of Dependency Confusion.\nWhy does Dependency Confusion work so effectively? # The success of a Dependency Confusion attack lies in the default configuration of many build systems and the way Maven resolves dependencies. There are several reasons why this attack vector is so effective:\n- Automatic prioritisation of public repositories - Trust the version number - Missing signature verification - Reliance on external code Typosquatting # Typosquatting is an attack technique that exploits user oversight by targeting common typos that can occur while typing package names in software development, such as in Maven. Attackers release packages with similar or slightly misspelt names that closely resemble legitimate libraries. When developers accidentally enter the wrong package name in their dependency definitions or automated tools resolve these packages, they download the malicious package. Typosquatting is one of the most well-known attack methods for manipulating package managers, such as Maven, npm, PyPI, and others that host publicly available libraries.\nBasics of typosquatting # Typosquatting is based on the idea that users often make typos when entering commands or package names. Attackers exploit this by creating packages or artifacts with names that are very similar to well-known and widely used libraries but differ in small details. These details may include minor variations such as missing letters, additional characters, or alternative spellings.\nTypical typosquatting techniques # Misspelled package names :\nOne of the most straightforward techniques is to change or add a letter in the name of a well-known library. An example would be the package **com.google.common** , which is often used. An attacker could use a package named **com.gooogle.common** (with an extra \u0026ldquo;o\u0026rdquo;) that is easily overlooked.\nDifferent spellings :\nAttackers can also use alternative spellings of well-known libraries or names. For example, an attacker could use a package named **com.apache.loggin** to publish the popular **com.apache.logging** looks similar, but due to the missing letter combination \u0026ldquo;g \u0026quot; and \u0026ldquo;n \u0026quot; at \u0026ldquo;logging \u0026quot; is easily overlooked.\nUse of prefixes or suffixes :\nAnother option is to add prefixes or suffixes that increase the similarity to legitimate packages. For example, an attacker could use the package **com.google.common-utils** or **com.google. commonx to publish** the same as the legitimate package**com.google.common** resembles.\nSimilarity in naming :\nAttackers can also take advantage of naming conventions in the open-source community by publishing packages containing common terms or abbreviations often used in combination with other libraries. An example would be releasing a package like **common-lang3-utils** , which is linked to the popular Apache Commons library `commons-lang3 \u0026rsquo; remembers.\nDangers of typosquatting # The threat of typosquatting is grave because it is difficult to detect. Developers often rely on their build tools like Maven to reliably download and integrate packages into their projects. If an incorrect package name is entered, you may not immediately realise that you have included a malicious dependency. Typosquatting is a form of social engineering because it exploits people\u0026rsquo;s susceptibility to errors.\nA successful typosquatting attack can lead to severe consequences:\nData loss Malware injection Loss of trust Maven typosquatting cases # There have also been incidents of typosquatting in the Maven community. In one case, a package named **commons-loggin** was published, corresponding to the legitimate Apache Commons logging package **commons-logging** . Developers who entered the package name incorrectly downloaded and integrated the malicious package into their projects, creating potential security risks.\nTyposquatting is a sophisticated and difficult-to-detect attack method that targets human error. Attackers take advantage of the widespread use of package managers such as Maven, npm, and PyPI by publishing slightly misspelt or similar-sounding packages that contain malicious code. Developers and organisations must be aware of this threat and take appropriate protective measures to ensure that only legitimate and trustworthy packages are included in their projects.\nProcess of a cache poisoning attack on Maven # A typical sequence of a cache poisoning attack on Maven could look like this:\nIdentification of a target repository : The attacker is looking for a Maven repository used by developers but may have vulnerabilities. This can happen, for example, through outdated versions of publicly available repository management tools.\nHandling Artifacts : The attacker manipulates the artefact, e.g. a JAR file, by adding malicious code. This can range from simple backdoors to complex Trojans.\nProvision of the poisoned artefact : The manipulated artefact is either uploaded to the public repository (e.g., in the form of a typosquatting package) or injected directly into a compromised target repository.\nDownload by developer : The developer uses Maven to update or reload the dependencies for his project. Maven downloads the poisoned artefact, which is stored in the local cache.\nCompromising the project : Maven will use the poisoned artefact from the cache in future builds. This artefact can then execute malicious code in the application\u0026rsquo;s context, resulting in system compromise.\nSecurity mechanisms to protect against cache poisoning # Various measures should be implemented on both the developer and repository provider sides to protect against cache poisoning attacks.\nRegular updates and patch management # Make sure Maven, its plugins, and all repository management tools are always up to date. Security updates should be applied immediately to address known vulnerabilities.\nUsing HTTPS # The use of encrypted connections (HTTPS) between Maven and the repository is crucial to ensure that no man-in-the-middle attacks can be performed on the transferring artifacts. Maven Central enforces HTTPS connections, but private repositories should also adhere to this standard.\nSignature verification # Another protective measure is the use of cryptographic signatures for artefacts. Maven supports the use of PGP signatures to ensure the integrity of artefacts. Developers should ensure that the signatures of downloaded artefacts are verified to ensure that they have not been tampered with.\nImprove repository security # Repository providers should ensure that their repositories are well protected by implementing robust authentication mechanisms, regular patches and updates to repository management tools such as Nexus or Artifactory.\nDependency Scanning and Monitoring # Tools like OWASP Dependency Check or Snyk can scan known dependency vulnerabilities. These tools can help identify malicious or stale dependencies and prevent them from entering the Maven cache.\nVersion Pinning # \u0026ldquo;Version pinning\u0026rdquo; means setting specific versions of dependencies in the pom.xml file instead of using dynamic version ranges ([1.0,)). This helps prevent unexpected updates and ensures that only explicitly defined versions of the artefacts are used.\nPrivate Maven-Repositories # One approach to maintaining control over dependencies is maintaining a private Maven repository within the organisation. This ensures that only checked artifacts end up in the internal cache, reducing the risk of introducing malicious dependencies into the build process.\nImplementation of code reviews and security checks # Conduct regular code reviews to ensure only trusted dependencies are used in projects. Automated security checks can provide additional security.\nUnderstanding CVEs in the context of Maven and cache poisoning # CVE (Common Vulnerabilities and Exposures) is a standardised system for identifying and cataloguing security vulnerabilities in software. Each CVE number refers to a specific vulnerability discovered and documented by security experts.\nThere are no specific CVEs that exclusively target cache poisoning in the context of Maven and cache poisoning. Instead, various vulnerabilities in Maven itself, its plugins, or the underlying repository management tools (such as Sonatype Nexus or JFrog Artifactory) can be indirectly exploited for cache poisoning attacks. These vulnerabilities could allow attackers to manipulate dependencies, compromise the integrity of downloads, or bypass Maven\u0026rsquo;s security mechanisms.\nRelevant CVEs related to Maven and repository management tools # Although no CVEs are explicitly classified as cache poisoning, there are several vulnerabilities in Maven and related tools that could potentially be exploited for cache poisoning attacks:\nCVE-2020-13949: Remote Code Execution in Apache Maven # Description : This vulnerability affected Maven Surefire plugin versions prior to 2.22.2, which could enable remote code execution (RCE). An attacker could use specially crafted POM files to execute malicious code on the build system. Relevance to cache poisoning : By running RCE, an attacker could inject crafted dependencies into the local Maven cache, which could compromise future builds. Reference : CVE-2020-13949 CVE-2021-25329: Denial of Service in Maven Artifact Resolver # Description : This vulnerability affected the Apache Maven Artifact Resolver component, which is responsible for resolving and downloading artefacts. A specially crafted POM file could lead to a denial of service (DoS). Relevance to cache poisoning : A DoS attack could impact the availability of Maven repositories, forcing developers to use alternative (possibly insecure) repositories, increasing the risk of cache poisoning. Reference : CVE-2021-25329 CVE-2019-0221: Directory Traversal in Sonatype Nexus Repository Manager # Description : This vulnerability allows attackers to access files within the Nexus Repository Manager through a directory traversal attack. Relevance to cache poisoning : By accessing critical files or configurations, attackers could compromise the repository manager and insert malicious artefacts, which are then downloaded by developers and stored in the local Maven cache. Reference : CVE-2019-0221 CVE-2022-26134: Arbitrary File Upload in JFrog Artifactory # Description : This vulnerability allowed attackers to upload arbitrary files to a JFrog Artifactory server, which could result in a complete compromise of the server. Relevance to cache poisoning : By uploading arbitrary files, attackers could inject malicious Maven artefacts into the repository, which developers then download and cache. Reference : CVE-2022-26134 CVE-2021-44228: Log4Shell (Log4j) # Description : This widespread vulnerability affected the Log4j library and allowed remote code execution by exploiting JNDI injections. Relevance to cache poisoning : Many Maven projects use Log4j as a dependency. A manipulated version of this dependency could enable RCE through cache poisoning. Reference : CVE-2021-44228 Analysis and impact of the mentioned CVEs # The above CVEs illustrate how vulnerabilities in Maven and related tools can potentially be exploited for cache poisoning and other types of attacks:\nRemote Code Execution (RCE) : Vulnerabilities such as CVE-2020-13949 and CVE-2021-44228 allow attackers to execute malicious code on the build system. This can be used to inject manipulated dependencies into the local cache. Denial of Service (DoS) : CVE-2021-25329 demonstrates how attackers can impact the availability of Maven repositories. This may result in developers being forced to resort to alternative sources that may be unsafe. Directory Traversal and Arbitrary File Upload : CVE-2019-0221 and CVE-2022-26134 demonstrate how attackers can compromise repository management tools to upload malicious artefacts that developers then unknowingly use in their projects. Future challenges and continuous security improvements # As the use of open-source libraries in modern software development continues to increase, the threat of attacks such as cache poisoning also increases. Automating the build process and relying on external repositories creates an expanded attack surface that attackers seek to exploit.\nIt is becoming increasingly important for companies and developers to cultivate a security culture prioritising the safe handling of dependencies and artefacts. This requires not only the implementation of technical protection measures but also the training of developers to raise awareness of the risks of cache poisoning and other dependency attacks.\nConclusion # Cache poisoning attacks on Maven caches are a severe risk, especially at a time when open-source components play an essential role in software development. The attacks exploit vulnerabilities in how dependencies are resolved, downloaded and cached.\nDevelopers and companies must implement best security practices to prevent such attacks. This includes using HTTPS, signing and verifying artefacts, securing repositories, regular vulnerability scanning, and controlling the dependencies introduced into their projects.\nAwareness of the risks and implementing appropriate security measures are key to preventing cache poisoning attacks and ensuring the integrity of software development processes.\nHappy Coding\nSven\n","date":"13 November 2024","externalUrl":null,"permalink":"/posts/cache-poisoning-attacks-on-dependency-management-systems-like-maven/","section":"Posts","summary":"Cache poisoning on Maven Caches is a specific attack that targets how Maven Caches manages packages and dependencies in a software development process. It’s essential to understand how Maven works before we look at the details of cache poisoning.\n","title":"Cache Poisoning Attacks on Dependency Management Systems like Maven","type":"posts"},{"content":"","date":"13 November 2024","externalUrl":null,"permalink":"/tags/code-injection-attack/","section":"Tags","summary":"","title":"Code Injection Attack","type":"tags"},{"content":"","date":"13 November 2024","externalUrl":null,"permalink":"/tags/dependency-management/","section":"Tags","summary":"","title":"Dependency Management","type":"tags"},{"content":"","date":"13 November 2024","externalUrl":null,"permalink":"/tags/maven/","section":"Tags","summary":"","title":"Maven","type":"tags"},{"content":"","date":"13 November 2024","externalUrl":null,"permalink":"/categories/tools/","section":"Categories","summary":"","title":"Tools","type":"categories"},{"content":"The annotations @Test and @Testable have played an important role in the Java ecosystem regarding application testing. They are essential tools that help developers make unit and automated testing more effective. In this paper, we will explore the differences and connections between @Test and @Testable analyze and the motivation behind the introduction of @Testable understand. We will also play the role of @Testable in developing your test engines and discuss their importance for the flexibility and expandability of tests.\n@Test and their usage @Testable - What is it for? The connection between @Test and @Testable The role of @Testable for developing your own TestEngines Conclusion @Test and their usage # Die Annotation @Test is familiar to most Java developers involved in unit testing. It comes from the JUnit framework, one of Java\u0026rsquo;s most commonly used frameworks for automated testing. @Test is used to identify methods as test methods that are then executed by the testing framework. When using JUnit 4 or 5, developers mark a method with @Testto, explicitly indicating that this method represents a test method to be tested. The JUnit framework takes over the execution of these methods and evaluates whether the expected conditions are met or errors occur. The main advantage of using @Test is that the framework automatically recognises the test methods and reports their results, including errors or successful runs. This makes maintaining and expanding tests easier, as all @Test-annotated methods can be executed without manual configuration.\nAn example of a simple test method looks like this:\nimport org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.assertEquals; public class CalculatorTest { @Test void testAddition() { Calculator calc = new Calculator(); assertEquals(5, calc.add(2, 3)); } } In this example, the method testAddition is paired with @Test, so the JUnit framework recognises it as a test method. Executing the test checks whether the addition works correctly. The goal is to ensure that the tested code exhibits the desired behaviour, which increases the application\u0026rsquo;s reliability.\nThe use of @Test makes the testing process easy and efficient. Developers can create many test methods automatically recognised and executed by the JUnit framework. This is particularly useful in large projects where test coverage is essential in ensuring that the codebase remains stable and bug-free. @Test allows developers to quickly receive feedback on the health of their applications, resulting in shorter development cycles and improved software quality.\n@Testable - What is it for? # @Testable comes from the package org.junit.platform.commons.annotation and was developed specifically for the JUnit 5 platform. The annotation @Testable is used to mark test methods that should be recognised by runtime environments such as test engines. The annotation @Test in JUnit 5 is a kind of special case @Testable-Annotation. In JUnit 5 is @Test yourself with @Testable annotated. This means that everyone with @Test marked method implicitly to @Testable is what ensures detection by testing engines. The use of @Testable in the annotation @Test ensures that all JUnit testing methods are automatically accessible to the testing platform without the need for additional annotations. This increases testability in different test environments and supports the modularity of JUnit 5.\nThe introduction of @Testable brought more flexibility to the test system. While @Test mainly specifies the concrete implementation of the test, @Testable is a common annotation that enables other testing engines to discover and execute test methods dynamically. This is particularly useful for facilitating integration between different testing tools and improving the extensibility of the JUnit 5 framework. Developers who need to perform different types of testing in a project benefit from the flexibility of @Testable because it enables different test engines to work together.\nAn example of using @Testable is rarely found at the developer level because the JUnit engine @Testable is used internally by all, with @Test automatically marking annotated methods as testable. This means developers usually don\u0026rsquo;t explicitly care about the annotation @Testable, as the runtime environment and the test engine handle this. This abstraction is advantageous because it saves developers the effort of explicitly ensuring their testing methods are discoverable - the platform does that for them.\nThe connection between @Test and @Testable # The main difference between @Test and @Testable lies in their application and purpose. While @Test is the specific testing method for the JUnit framework @Testable a general meta-annotation that helps test runtime environments better recognise and process test methods. @Testable provides an abstraction layer that ensures that any method marked as a test is also available to the testing engine. This mainly helps when developing frameworks and tools based on the JUnit platform.\nJUnit 5 is a significant evolution compared to JUnit 4. It uses a more modular architecture and a testing engine called the \u0026ldquo;JUnit Platform\u0026rdquo;. This platform is designed to allow tests to be executed more flexibly so that other testing frameworks can also use the platform. @Testable plays an important role in this context as it ensures the visibility of the tests on the platform. Developers benefit from a simplified integration of test tools, as the platform contains all tests @Testable that are annotated, recognised and executed regardless of the test framework used.\nThe introduction of @Testable is, therefore, in the context of JUnit 5\u0026rsquo;s extensibility and flexibility. While JUnit 4 was primarily designed for unit testing, JUnit 5 enables better integration with other testing and tools. This makes the testing infrastructure for modern Java applications more powerful and adaptable. Using different testing engines and frameworks on the same platform significantly improves application testability and maintainability.\nThe role of @Testable for developing your own TestEngines # Die Annotation @Testable is crucial for developers who want to develop their testing engines. It provides a standardised way to mark test methods for the JUnit Platform, meaning that test engines compatible with it can recognise and execute test methods without making any special customisations. Using @Testable, Developers can ensure that their tests are recognised regardless of the testing framework used, promoting interoperability and extensibility of test engines.\nThe JUnit Platform is designed to support various testing engines, significantly increasing developers\u0026rsquo; flexibility. @Testable provides a type of \u0026ldquo;contract\u0026rdquo; that allows the JUnit Platform to identify and execute tests, whether they were built using JUnit, TestNG, or a custom testing engine. Developers can write their own test engines seamlessly integrated into the JUnit Platform and benefit from the existing functionalities. This does @Testable, a central building block for the development of an expandable and modular test environment.\nAnother advantage of using @Testable for your own test engines is the standardisation of the test process. By labelling all test methods with @Testable, Developers can ensure that their tests run consistently across different projects and teams. This not only promotes the reusability of test engines but also the ability to develop new test engines that meet specific requirements without disrupting the fundamental architecture of the JUnit Platform.\nWith @Testable It also becomes possible to advance the development of specialized testing engines optimized for specific types of testing. For example, performance testing, security testing or end-to-end testing could be carried out by purpose-built engines based on the JUnit Platform. This creates a modular and versatile testing environment that allows the best tools to be used for different testing needs without compromising integration and compatibility.\nConclusion # The annotations @Test and @Testable play different but complementary roles in the Java testing ecosystem. @Test identifies specific testing methods for JUnit, while @Testable ensures that these methods are accessible to testing engines on a more general level. The introduction of @Testable as part of JUnit 5 offers greater flexibility and makes it easier to expand the testing system, which meets the modern requirements for testing in the Java world.\nOverall, both annotations show the continuous development of testing tools in Java to meet the growing requirements for maintainability, extensibility and integration. The introduction of @Testable highlights the trend towards a more flexible and expandable testing infrastructure that makes it possible to combine different test types and engines in a common platform. This allows developers to make testing more efficient and test their applications comprehensively and reliably. By combining @Test and @Testable, A robust test basis is created that supports the development process and sustainably improves the quality of software products.\n","date":"8 November 2024","externalUrl":null,"permalink":"/posts/junit-annotations-in-focus-the-connection-between-test-and-testable/","section":"Posts","summary":"The annotations @Test and @Testable have played an important role in the Java ecosystem regarding application testing. They are essential tools that help developers make unit and automated testing more effective. In this paper, we will explore the differences and connections between @Test and @Testable analyze and the motivation behind the introduction of @Testable understand. We will also play the role of @Testable in developing your test engines and discuss their importance for the flexibility and expandability of tests.\n","title":"JUnit annotations in focus: The connection between @Test and @Testable","type":"posts"},{"content":"","date":"8 November 2024","externalUrl":null,"permalink":"/categories/junit5/","section":"Categories","summary":"","title":"JUnit5","type":"categories"},{"content":"","date":"8 November 2024","externalUrl":null,"permalink":"/tags/junit5/","section":"Tags","summary":"","title":"JUnit5","type":"tags"},{"content":"","date":"8 November 2024","externalUrl":null,"permalink":"/tags/tdd/","section":"Tags","summary":"","title":"Tdd","type":"tags"},{"content":"","date":"8 November 2024","externalUrl":null,"permalink":"/categories/tdd/","section":"Categories","summary":"","title":"TDD","type":"categories"},{"content":"","date":"8 November 2024","externalUrl":null,"permalink":"/tags/test-engine/","section":"Tags","summary":"","title":"Test Engine","type":"tags"},{"content":" Screenshot\nDescription:\n\u0026ldquo;JUnit5 - Effective testing\u0026rdquo;\nTarget group : This workshop is aimed at Java developers who want to expand their unit testing skills and software testers who want to take Java application testing to a higher level. The workshop is also ideal for development teams that want to introduce a uniform and professional testing strategy to increase software quality. This workshop also offers an excellent introduction for anyone looking for a comprehensive introduction to the concepts and practices of testing with JUnit 5. Both beginners and developers with advanced knowledge benefit from the practical examples and exercises to develop a deep understanding of the possibilities of JUnit 5.\nCourse requirements : Basic knowledge of Java programming is required. Experience using development tools such as IDEs (e.g., IntelliJ, Eclipse) is advantageous but not absolutely necessary.\nCourse Description :\nWelcome to our intensive two-day workshop, “Effective Testing with JUnit 5”, where you will learn systematically and step by step how to use JUnit\u0026rsquo;s latest features and best practices to create reliable and maintainable unit tests for your Java applications. In this workshop, each concept is explained in detail, supported by practical examples, and reinforced with exercises to apply what you have learned immediately. This workshop is aimed at both beginners who want to build a solid foundation in Java application testing and experienced developers who want to expand their knowledge of the advanced features and efficient use of JUnit 5 to ensure high-quality software, making development cycles more efficient.\nUnit tests are a fundamental building block of the agile development process and a crucial factor for the quality and stability of your software. They make it possible to detect errors early and increase the software\u0026rsquo;s maintainability, which can reduce development costs in the long term. JUnit is the most widely used framework for testing Java applications, giving you all the tools you need to develop high-quality tests. With JUnit5 you have a modern and powerful platform that offers numerous advantages, from more flexible test configurations to improved options for parameterisation and integration. In this workshop, you will learn how to use the new functions of JUnit 5 optimally and how you can effectively integrate them into your development processes to ensure and increase the quality of your software in the long term. You will learn how to design tests to be easy to understand, maintain, and reusable, which is particularly beneficial in larger projects.\nLearning objectives :\nAfter completing the workshop, participants will be able to:\nUnderstand the differences and advantages of JUnit 5 compared to previous versions. Write effective unit tests and apply best practices in test development. To understand the JUnit 5 architecture and the role of Platform, Jupiter and Vintage modules. Leverage advanced features such as dynamic testing, custom tags, conditional execution, and test set reuse. Integrate JUnit 5 with common development tools and build systems like Maven and Gradle. Course content :\nIntroduction to JUnit and the importance of unit testing Basic concepts of testing in software development Why JUnit? Role of testing in agile development Test Driven Development (TDD) with JUnit The architecture of JUnit 5 Divided into modules: Platform, Jupiter and Vintage How modularising JUnit 5 improves testability and extensibility Differences between JUnit 4 and JUnit 5 – what’s new? Basics of testing with JUnit 5 Setting up the project with Maven and Gradle “@DisplayName” – Human-readable test names for better understandability Test-Lifecycle-Annotierungen („@BeforeEach“, „@AfterEach“, „@BeforeAll“, „@AfterAll“) Assertions (“Assertions.assertEquals()”, “assertTrue()”, etc.) and their extensions Advanced features Parameterised testing : Using @ParameterizedTest for more flexibility and reusability Test factories and dynamic tests : “@TestFactory” for creating dynamic tests Conditional tests : Run tests based on conditions (“@EnabledOnOs”, “@DisabledIf”, etc.) Tagging and filtering tests : Using @Tag to group and selectively run tests Extension of JUnit 5 Write your extensions : Using Extension to customise test behaviour Third-party extensions : Integration of libraries like Mockito to create mocks and stubs Best practices and testing strategies Write maintainable tests : Building readable and reusable test classes Avoid sources of error : Common pitfalls and problems when developing unit tests Test Driven Development (TDD) : Practical approaches and examples for applying TDD with JUnit 5 Practical exercises and examples Step-by-step instructions to set up a new JUnit 5 project Code examples : Hands-on with practical examples of typical test scenarios Team exercises and collaboration : Working together in groups to solve complex test tasks and deepen what you have learned Independent exercises : Opportunity to directly apply what you have learned and develop your test cases Teaching methods :\nThe workshop includes interactive lectures, live demos, group discussions and practical exercises. Participants are actively involved in learning by writing their test cases, implementing best practices in realistic scenarios and developing solutions together. Interactive quizzes and tasks are available to deepen understanding and ensure learning progress. Participants benefit from direct interaction with the course leader and the opportunity to ask questions in real-time. Why should you attend this workshop?\nTesting is an indispensable part of software development. By learning JUnit 5, you will improve the quality of your applications and your ability to develop more efficiently and safely. JUnit 5 offers many possibilities that help beginners and experienced developers expand their testing know-how. In this workshop, you will learn practically how to use the latest features of JUnit 5 in your development process to increase your software\u0026rsquo;s quality significantly. Direct exchange with the course instructor and other participants also offers yo test strategies and best practices.\nBenefits for participants :\nPractical approach: We offer many practical examples and exercises so that you can immediately apply what you have learned. Flexible Format: Choose between on-site or online participation that best suits your needs. Intensive exchange: Benefit from the opportunity to ask questions directly and work on solutions with other participants. Flexibility and modularity : Use JUnit 5\u0026rsquo;s modular architecture to better organise and customise your tests. Certification :\nAt the end of the workshop, all participants receive a participation certificate confirming the knowledge they have acquired in ​​JUnit5.\nConclusion :\nWith JUnit5, you get a powerful tool that makes your tests more maintainable and meaningful. In this workshop, you will learn how to use the full potential of JUnit 5 to develop high-quality software. Benefit from our comprehensive introduction to the world of unit testing and take the next step in your career as a Java developer.\nRegistration :\nInterested? Register today for the intensive two-day workshop—either online or on-site—and begin your journey into the world of automated testing with JUnit5!\nLanguage :\nGerman or English\nWorkshop Duration:\n2 days\nLocation\nIn house or remote\n","date":"6 November 2024","externalUrl":null,"permalink":"/workshop-junit5-effective-testing-of-java-applications/","section":"Workshop - JUnit5 - Effective testing of Java applications","summary":" Screenshot\nDescription:\n“JUnit5 - Effective testing”\nTarget group : This workshop is aimed at Java developers who want to expand their unit testing skills and software testers who want to take Java application testing to a higher level. The workshop is also ideal for development teams that want to introduce a uniform and professional testing strategy to increase software quality. This workshop also offers an excellent introduction for anyone looking for a comprehensive introduction to the concepts and practices of testing with JUnit 5. Both beginners and developers with advanced knowledge benefit from the practical examples and exercises to develop a deep understanding of the possibilities of JUnit 5.\n","title":"Workshop - JUnit5 - Effective testing of Java applications","type":"workshop-junit5-effective-testing-of-java-applications"},{"content":"","date":"5 November 2024","externalUrl":null,"permalink":"/tags/mocking/","section":"Tags","summary":"","title":"Mocking","type":"tags"},{"content":"Extensive use of mocking frameworks such as Mockito in software development can lead to unrealistic tests. This is because mocking frameworks simulate dependencies of classes or methods in order to test them in isolation. However, when too many mock objects are used, the test often loses touch with reality, which can affect the validity and reliability of the tests. It is important to use mocking carefully to find the right balance between isolated testing and realistic simulation.\nA mock is a simulated object that imitates the behaviour of a real component without executing any actual logic. This is particularly useful when a dependency is difficult to test, for example, because it involves an external database or web service that may have states that are difficult to reproduce or where testing would be very time-consuming and costly. Mocking creates a simplified and manageable version of this dependency, allowing targeted and isolated unit testing to be carried out. Mocking can be a precious technique in these cases, especially for writing independent tests that do not require external infrastructure. It reduces the complexity of tests and makes them faster and more deterministic.\nHowever, it becomes problematic when the logic of the tested class is heavily influenced by these dependencies, especially when these dependencies have complex logic or dynamic behaviour. For example, a class that reads data from a database might implement logic that responds to the available data contents. This dynamic nature often cannot be realistically replicated with mocks, as mock objects are usually limited to predefined, simple return values. As a result, the test can lose touch with the objective complexity and only inadequately reflect the actual application logic. Realistic tests are created that only consider some possible application behaviours in interaction with other components, resulting in incomplete test coverage.\nExtensive use of mocks often results in tests being tailored very specifically to the simulated scenarios. In such cases, the tests become helpful only for the implemented interfaces and the programmed code rather than for the actual use cases, which may contain a variety of unexpected events and uncontrolled factors. This can lead to a false sense of security that the application is functioning correctly, even though essential real-world conditions still need to be considered. For example, network failures, errors in data transmission or inconsistent data sets could not be simulated in the mocks because the tests focus only on positive confirmation of the function. This means that critical errors that could occur in a natural environment are missed, leading to severe problems in the production environment. To combat this, mocking should be used judiciously, and tests should also be performed that incorporate real-world dependencies to ensure that the code works correctly under real-world conditions such as network problems, database failures, or changing data.\nA concrete example of this would be an application that retrieves data from a REST API and processes it. In a typical scenario, this REST API could return different data sets, ranging from simple JSON responses to complex, nested structures. When you replace the REST API with a mock object, you only simulate a specific reaction, which is highly simplified and mostly represents the \u0026lsquo;happy path\u0026rsquo;. In reality, however, the REST API could react in various ways: data could be missing, the data structure could change, new fields could be added, or network problems and unexpected error messages could occur. Additionally, errors such as receiving an empty response or submitting an HTTP error (e.g. 500 or 404) are often problematic to represent when only a mock is used. Tests based only on mocks cannot adequately simulate these realities because they need to consider the multitude of possible scenarios and failure cases that can arise in actual operation. This leads to a false sense of security about the robustness of the code. If the code is only tested in a strictly controlled environment, without considering the real variability of the API and its potential failures, serious errors may appear in the production environment that were never previously detected. Using accurate integration tests that respond to actual API responses and errors would be much more realistic in this case and would help ensure application robustness under real-world conditions.\nAnother problem arises when the extensive use of mocks oversimplifies the dependencies. For example, consider an e-commerce application that performs inventory checking. In a real-world implementation, this inventory check can involve a variety of complex operations, such as synchronising inventory across multiple geographically distributed warehouses, honouring reservations for ongoing orders, or updating inventory in real-time. These tasks require sophisticated logic, potentially involving numerous systems and technologies, to ensure inventories are always accurate. When you fully mock a warehouse management class, these complex dependencies are replaced with simple, simulated return values. However, this means that the test loses connection to the real functioning of the system. There is no guarantee that synchronisation logic will work smoothly under real-world conditions, such as network delays or inconsistencies between warehouses. The complexity of the real system is never taken into account in the tests, so possible sources of error remain undetected. In a production environment, unforeseen problems could arise that could have been avoided with a more realistic test environment. These complex scenarios show that real integration tests are necessary to ensure that the interaction between the components works correctly, even under load conditions or in the event of errors in the dependencies.\nAnother problem arises when test implementations are created too specifically for the mock scenarios. It often happens that developers unconsciously tailor the mock configuration exactly to the logic being tested without taking into account the realistic spread of the data or the potential errors. This results in the tests not reflecting unforeseen events that may occur in a real environment. An example of this is a database query where the test always assumes a successful query because the mock was configured that way. The realistic error case of a database failure, a zero response or a timeout is not practised. This means that the corresponding error handling is noticed in the production environment once it has already had an impact, which can lead to critical problems. Such gaps in error handling could be avoided if tests included a more realistic and varied simulation of possible scenarios. Therefore, tests must also cover hostile and unpredictable scenarios to ensure the application is robust enough to handle errors and exceptions.\nAnother problem when using mocking frameworks is that the functionality of the mocks only sometimes follows the actual implementation. Often, a simplified or even incorrect replica of the real components is created, which only reflects some aspects of real behaviour. For example, when replicating an external service using a mock, certain side effects, state changes, or dependencies in the real implementation cannot be adequately considered. This can cause the test to pass even though the actual implementation would fail due to its complexity.\nLet\u0026rsquo;s take an example of a Java application that accesses an external cache service to cache data. Suppose we have a class ProductService that reads product information from a database and stores it in a cache to improve performance. In a unit test, the cache service method could easily be mocked to ensure the put() call runs correctly. The corresponding Java code could look like this:\nimport static org.mockito.Mockito.*; public class ProductServiceTest { @Test public void testFetchProduct() { CacheService cacheService = mock(CacheService.class); DatabaseService databaseService = mock(DatabaseService.class); ProductService productService = new ProductService(databaseService, cacheService); Product product = new Product(\u0026#34;123\u0026#34;, \u0026#34;TestProduct\u0026#34;); when(databaseService.getProduct(\u0026#34;123\u0026#34;)).thenReturn(product); productService.fetchProduct(\u0026#34;123\u0026#34;); verify(cacheService, times(1)).put(\u0026#34;123\u0026#34;, product); } } This test verifies that the product is cached correctly. However, it is not tested here whether the cache mechanism really works as desired, e.g., whether the cache is actually used for repeated requests to avoid unnecessary database queries. Such an error could go undetected in production if, in reality, the cache service is not functioning properly due to a synchronisation issue or misconfiguration. In addition, concurrent accesses to the cache could lead to synchronisation problems that need to be taken into account by simple mocking.\nThis test, therefore, provides no guarantee that the cache logic will work correctly under real-world conditions such as high load or concurrent accesses. The mocks only represent a simplified version of reality without taking into account all the possible scenarios that could occur in a real environment. Therefore, in addition to unit tests that use mocks, it is crucial also to perform integration tests that ensure that the cache service works correctly under realistic conditions. A classic example would be a component that internally uses caching mechanisms to minimise the number of requests to an external service. A simple mock could ignore the caching mechanism, so the test only checks whether a request is sent correctly to the external service, but not whether the caching logic works as intended. This allows cache management bugs or inefficient implementations to go undetected. Mocks also often do not realistically represent competing accesses to a resource, which could lead to synchronisation problems in the real implementation. Therefore, it is essential to ensure that mocks are as close as possible to the actual implementation to ensure realistic testing scenarios.\nAn alternative to mocking is integration or end-to-end testing, which tests the actual components and dependencies. Integration testing helps ensure that the application\u0026rsquo;s various modules interact with each other as expected, while end-to-end testing verifies the overall system under realistic conditions. This does not use mocks but rather real databases, REST APIs and other services, which better simulate the complexity of the real environment.\nAnother alternative is so-called \u0026lsquo;contract testing\u0026rsquo;. Contract testing checks whether communication between two services occurs according to a defined specification. This allows both sides of the communication to be developed and tested independently without simulating the entire service. This ensures a certain level of security that both components work correctly with each other without resorting to extensive mocking.\nIn addition, test containers or in-memory databases can be used as alternatives to mocks, especially for database testing. Using tools like Testcontainers, real database instances can be deployed in Docker containers for testing so that real database logic is tested. In-memory databases like H2 in Java are also useful for making testing more realistic because they simulate actual data logic but without the cost and complexity of a real database connection.\nAnother alternative method is the use of so-called \u0026lsquo;fake\u0026rsquo; objects. Unlike mocks, which only simulate behaviour, fakes implement a reduced version of the real logic. These fakes are not as complex as the real components but still offer more realism than simple mocks. An example of this would be a \u0026lsquo;fake\u0026rsquo; database object that mimics basic storage operations in a simple data structure rather than actually connecting to a database. Fakes are particularly useful when you need a realistic but not fully functional, environment to test certain aspects of the application. They are more flexible than mocks and provide the ability to run through different scenarios without simulating the full complexity of real implementations.\nIn summary, excessive use of mocking frameworks like Mockito often results in tests not being realistic. The isolation of components enabled by mocking is useful for performing isolated unit testing. However, if too many aspects of the application are simulated, the tests lose the ability to represent the interaction of real components and the actual challenges that might arise in operation. There is then a risk that the software in real operation will not show the behaviour that the tests suggest. It, therefore makes sense to find a balance between mocking and integration tests in order to represent realistic scenarios adequately. At the same time, other techniques, such as contract testing or the use of fakes, should also be considered to ensure that the tests replicate the real environment as closely as possible, thus ensuring a robust and reliable application.\nHappy Coding\nSven\n","date":"5 November 2024","externalUrl":null,"permalink":"/posts/the-risks-of-mocking-frameworks-how-too-much-mocking-leads-to-unrealistic-tests/","section":"Posts","summary":"Extensive use of mocking frameworks such as Mockito in software development can lead to unrealistic tests. This is because mocking frameworks simulate dependencies of classes or methods in order to test them in isolation. However, when too many mock objects are used, the test often loses touch with reality, which can affect the validity and reliability of the tests. It is important to use mocking carefully to find the right balance between isolated testing and realistic simulation.\n","title":"The Risks of Mocking Frameworks: How Too Much Mocking Leads to Unrealistic Tests","type":"posts"},{"content":"","date":"4 November 2024","externalUrl":null,"permalink":"/tags/cwe-1007/","section":"Tags","summary":"","title":"CWE-1007","type":"tags"},{"content":"","date":"4 November 2024","externalUrl":null,"permalink":"/tags/homoglyphs/","section":"Tags","summary":"","title":"Homoglyphs","type":"tags"},{"content":"The world of cybersecurity is full of threats, many of which are surprisingly subtle and challenging to detect. One such threat is the problem of so-called homoglyphs. CWE-1007, also known as \u0026ldquo;Insufficient Visual Distinction of Homoglyphs Presented to User\u0026rdquo;, is a vulnerability often used by attackers to deceive and compromise your systems or data. In this blog article, you will get a deep insight into CWE-1007, understand its mechanisms, and how to protect yourself from such attacks. We will discuss examples, technical challenges, and best practices that can help you as a developer understand and mitigate this threat.\nWhat are Homoglyphs? CWE-1007: A detailed investigation Examples for CWE-1007 Why is CWE-1007 dangerous? Possible attack scenarios Defence strategies against CWE-1007 Technical challenges in homoglyph recognition Best practices for developers Case Study: Homoglyph Domain Attack Homoglyph recognition tools Which CWEs are often used in conjunction with CWE-1007? 1. CWE-601: Open Redirect: 2. CWE-643: Improper Neutralization of Data within XPath Expressions: 3. CWE-79: Cross-Site Scripting (XSS): 4. CWE-20: Improper Input Validation: 5. CWE-77: Command Injection: Example application in Vaadin Flow Conclusion Next Steps What are Homoglyphs? # Before we delve into CWE-1007, we must understand what homoglyphs are. Homoglyphs are characters that look visually similar but represent different Unicode codes. This can involve different letters, numbers or symbols. A well-known example is the Latin letter \u0026ldquo;O\u0026rdquo; and the number \u0026ldquo;0\u0026rdquo;, which can look almost identical to the human eye. There are many other examples, such as \u0026ldquo;l\u0026rdquo; (small L) and \u0026ldquo;I\u0026rdquo; (capital i), or Cyrillic letters that look like Latin letters.\nThe visual similarity of homoglyphs is often exploited to deceive you. Attackers use such characters to create phishing websites, generate fake URLs, or corrupt code to make you believe you are dealing with a trustworthy resource. This is particularly problematic because we humans use visual patterns to make quick decisions and can more easily fall victim to such deceptions.\nCWE-1007: A detailed investigation # CWE-1007 refers to the insufficient visual discrimination of homoglyphs when presented as a user. If a system cannot distinguish between similar-looking characters or alert you to them, this can lead to significant security risks. You could accidentally click on a malicious link, visit a fake domain, or trust a fraudulent command.\nThis vulnerability is particularly common when displaying URLs, usernames, or commands. For example, you could click on a URL that appears to be correct but actually uses homoglyphs to represent a fake website. This allows attackers to steal passwords, credit card information, or other sensitive data. This problem is compounded by widespread internet use, as you constantly interact with potentially harmful content.\nExamples for CWE-1007 # A typical example of this vulnerability is the use of fake domains that use homoglyphs. Let\u0026rsquo;s say you receive an email with a link to \u0026ldquo;paypa1.com\u0026rdquo; (which uses the number \u0026ldquo;1\u0026rdquo; instead of the letter \u0026ldquo;l\u0026rdquo;). Without careful consideration, you might think this is a legitimate link to the PayPal website. The same principle can also apply to usernames on social networks or even essential commands in a console.\nAnother example is the use of homoglyphs in source code. Attackers could insert fake characters into the code that look like legal characters but result in different functionality. This can be particularly dangerous in open-source projects or teams where multiple developers work on the same code and could miss similar characters. This leads to security vulnerabilities that can be exploited for attacks and poses a high risk to the integrity of the code.\nAn everyday example is a fake username on social networks. An attacker could create a username like \u0026ldquo;facebook_support\u0026rdquo; using the Cyrillic \u0026ldquo;е\u0026rdquo;, which looks visually similar to the Latin \u0026ldquo;e\u0026rdquo;. You might think it\u0026rsquo;s an official support channel and click on malicious links or reveal sensitive data.\nWhy is CWE-1007 dangerous? # CWE-1007 is dangerous because humans rely heavily on visual cues to make decisions. The human eye is trained to recognise patterns and process information quickly, but often without closely examining each character. Attackers can specifically exploit this weakness to deceive you.\nThe danger of this vulnerability is not only that you could fall for phishing attacks. It can also lead to more severe security breaches, such as stealing credentials, tampering with software, or inserting malicious code into seemingly legitimate projects. In addition, this deception can also cause financial damage because you trust fake websites and enter your payment information, which can then be misused.\nAnother danger is that companies can lose their reputation due to these vulnerabilities. If you repeatedly fall for counterfeit versions of a well-known brand, your trust in that brand can be severely affected. Attackers use this method to undermine user trust and specifically to maximise damage to companies.\nPossible attack scenarios # Phishing attacks using fake URLs : Attackers can create a URL almost identical to a known website. You click on the link without noticing that the domain has a slight change (e.g. a Cyrillic letter instead of a Latin letter). This allows attackers to access sensitive information such as passwords or credit card details.\nCode injection through fake characters : In complex software projects, homoglyphs can cause the code to execute differently than it seems at first glance. You or other developers may miss these signs, resulting in hard-to-find vulnerabilities in your code. This could be used to insert malicious functions only noticeable in the production environment.\nSocial engineering in social networks : An attacker could create a username almost identical to a trusted contact\u0026rsquo;s to deceive you and steal information. For example, someone could use the name \u0026ldquo;LinkedIn Support\u0026rdquo; with a slightly modified letter to trick you into giving up your login information or clicking on malicious links. The deception can be particularly effective if you don\u0026rsquo;t carry out additional security checks.\nDefence strategies against CWE-1007 # To protect you from the threat of CWE-1007, technical and organisational measures are necessary. Here are some strategies that can help you minimise risk:\nUnicode normalisation : One of the most effective methods of recognising homoglyphs is Unicode normalisation. Unicode normalisation converts similar-looking characters into standard forms, making identifying them more accessible. This can prevent different writing systems from being used to deceive.\nimport java.text.Normalizer; public class UnicodeNormalizationExample { public static void main(String[] args) { String suspiciousString = \u0026#34;päypäl.com\u0026#34;; String normalizedString = Normalizer.normalize(suspiciousString, Normalizer.Form.NFKC); System.out.println(\u0026#34;Normalized String: \u0026#34; + normalizedString); } } This example shows how characters can be normalised to ensure that visually similar but different characters are recognised as such.\nUser training : Training in dealing with suspicious emails, links and domain names is an essential line of defence. You should learn to check URLs carefully and be aware that characters from different writing systems can be used to deceive you. This training should include regular exercises and examples to increase awareness and improve your ability to recognise such attacks.\nSecurity warnings in the browser : Modern browsers have mechanisms to warn you if you visit a suspicious domain or use characters from different Unicode writing systems. These warnings should be enabled to protect you from such threats. Browser extension developers could develop additional filtering mechanisms that promptly alert you to possible deception.\nCode reviews and static code analysis tools : In software projects, developers should conduct code reviews to identify suspicious characters. Static code analysis tools can also help detect homoglyphs in code and mitigate potential security risks.\npublic class CodeReviewExample { public static boolean containsSuspiciousCharacters(String input) { // Checks whether the string contains non-Latin characters return !input.matches(\u0026#34;^[\\ -\\~]*$\u0026#34;); } public static void main(String[] args) { String input = \u0026#34;paypaı.com\u0026#34;; // contains the homoglyph \u0026#34;ı\u0026#34; (small i without a dot) if (containsSuspiciousCharacters(input)) { System.out.println(\u0026#34;Suspicious characters found: \u0026#34; + input); } } } This example shows how a simple check can help identify potentially dangerous signs and act accordingly.\nAllowing specific character sets : Another risk mitigation measure is limiting the use of certain characters. For example, an application might specify that only Latin characters are allowed in usernames or URL paths to reduce the risk of homoglyph attacks. This restriction helps reduce the attack surface.\npublic class CharacterWhitelistExample { public static boolean isValidInput(String input) { return input.matches(\u0026#34;^[a-zA-Z0-9]*$\u0026#34;); } public static void main(String[] args) { String username = \u0026#34;username\u0026#34;; // contains a non-Latin character if (isValidInput(username)) { System.out.println(\u0026#34;Username is valid.\u0026#34;); } else { System.out.println(\u0026#34;Username contains invalid characters.\u0026#34;); } } } This example shows a simple method for restricting the allowed characters in input to prevent using homoglyphs.\nTechnical challenges in homoglyph recognition # Recognising homoglyphs represents a significant technical challenge. One reason for this is the number of characters in the Unicode standard. Unicode includes thousands of characters from different writing systems that may look similar or identical. An algorithm intended to identify such characters must be able to distinguish between visual similarities and actual meaning.\nAnother problem is that not all applications or systems can display Unicode correctly. In some cases, different characters may appear identical through the rendering process, making it even more difficult for you to distinguish between legitimate and fake content. These technical challenges require the development of robust checking mechanisms that ensure that such attempts at deception can be detected.\nBest practices for developers # As a developer, you play a crucial role in preventing CWE-1007. Here are some best practices to consider when developing secure applications:\nInput validation : User input data should always be validated to ensure that no dangerous characters are used. If possible, only certain character sets should be allowed.\npublic class InputValidationExample { public static boolean isValidInput(String input) { return input.matches(\u0026#34;^[a-zA-Z0-9]*$\u0026#34;); } public static void main(String[] args) { String userInput = \u0026#34;hello123\u0026#34;; if (isValidInput(userInput)) { System.out.println(\u0026#34;Input is valid.\u0026#34;); } else { System.out.println(\u0026#34;Input contains invalid characters.\u0026#34;); } } } Escape and encode : Data used for display or transmission should always be escaped and encoded to ensure no harmful characters are inserted unnoticed.\nimport org.apache.commons.text.StringEscapeUtils; public class EscapeEncodeExample { public static void main(String[] args) { String userInput = \u0026#34;\u0026lt;script\u0026gt;alert(\u0026#39;XSS\u0026#39;);\u0026lt;/script\u0026gt;\u0026#34;; String escapedInput = StringEscapeUtils.escapeHtml4(userInput); System.out.println(\u0026#34;Escaped Input: \u0026#34; + escapedInput); } } This example shows how potentially malicious input can be correctly handled to prevent attacks such as Cross-Site Scripting (XSS).\nConscious use of fonts : Choosing the right font can help you recognise homoglyphs better. Some fonts distinguish more clearly between similar-looking characters, making it easier for you to spot differences. Examples of such fonts include \u0026lsquo;Consolas\u0026rsquo;, \u0026lsquo;Courier New\u0026rsquo;, and \u0026lsquo;DejaVu Sans Mono\u0026rsquo;. These fonts are handy when characters need to be clearly distinguished from each other, for example, in source code or security-related information.\nProvide additional contextual information : If you\u0026rsquo;re asked to review sensitive information like URLs or usernames, it can be helpful to provide additional contextual information to help verify legitimacy. This includes warnings or visual markers that alert you that certain signs may be potentially dangerous.\nCase Study: Homoglyph Domain Attack # A famous case study that illustrates the danger of homoglyphs is the attack via a fake “apple.com” domain. In this case, the attacker registered a domain that consisted of Cyrillic characters that visually looked identical to \u0026ldquo;apple.com.\u0026rdquo; When you clicked on this domain, you were taken to a website that looked almost exactly like the genuine Apple website. This allowed the attacker to steal sensitive information, such as login credentials.\nSuch attacks highlight the importance of paying attention to the visual distinction of characters when presenting information. Without careful examination, you would have no technical way of recognising the fake domain. Such attacks highlight the need to use technical and human defences to protect yourself from deception.\nHomoglyph recognition tools # There are some tools and libraries designed explicitly for homoglyph recognition. These tools can help you, as a developer and security professional identify and remediate potential vulnerabilities in your systems. The most well-known tools include:\nDNSTwist : A tool used to generate similar domain names and check whether they have been registered for phishing attacks. DNSTwist can help detect potential threats early and take countermeasures.\nHomoglyph Detector Libraries : Various program libraries recognise homoglyphs in strings. These libraries can be integrated into applications to ensure suspicious characters are detected. Such libraries are particularly useful when users provide security-relevant input.\nUnicode-Analyse-Tools : These tools can help check characters for their Unicode representation, ensuring that similar characters are not used unnoticed. Unicode analysis tools can help you identify and fix problems early.\nWhich CWEs are often used in conjunction with CWE-1007? # CWE-1007 is often used with other vulnerabilities that undermine the security and trustworthiness of applications. Here are some of the most common CWEs that frequently appear associated with CWE-1007:\nCWE-601: Open Redirect: # This vulnerability occurs when an application allows users to be redirected to a different, potentially dangerous URL without sufficient validation. CWE-1007 can be used here to deceive users by presenting a seemingly legitimate but homoglyph-manipulated URL redirecting to a malicious resource.\nCWE-643: Improper Neutralization of Data within XPath Expressions: # When user input is used in XPath queries without sufficient validation, CWE-1007 can allow attackers to use visually similar characters to manipulate the query and access data that would not usually be available.\nCWE-79: Cross-Site Scripting (XSS): # Cross-site scripting occurs when an application does not adequately escape or validate user input returned for display. CWE-1007 can also be dangerous here if homoglyphs are used to hide malicious scripts in a context that looks harmless to the user.\nCWE-20: Improper Input Validation: # This general vulnerability occurs when an application does not sufficiently validate user input. CWE-1007 is particularly problematic if no normalisation and validation measures are taken, as visually similar characters could pass as legitimate input.\nCWE-77: Command Injection: # Command injection occurs when a user\u0026rsquo;s input is used in a way that results in the execution of system commands. By using homoglyphs, attackers could manipulate input so that the actual command deviates from the expected execution, which could give an attacker control over the system.\nCombining CWE-1007 with these other vulnerabilities allows attackers to carry out significantly more complex and effective attacks. They leverage the visual deception of homoglyphs to exploit vulnerabilities that might be easier to detect without such deception.\nExample application in Vaadin Flow # To illustrate all the best practices discussed in a real application, here is an example application in Vaadin Flow. Vaadin Flow is a popular open-source Java framework for building modern web applications. This sample application shows how you can implement Unicode normalisation, input validation, escaping, and encoding methods in a secure web application to prevent homoglyph-based attacks.\nimport com.vaadin.flow.component.button.Button; import com.vaadin.flow.component.notification.Notification; import com.vaadin.flow.component.orderedlayout.VerticalLayout; import com.vaadin.flow.component.textfield.TextField; import com.vaadin.flow.router.Route; import org.apache.commons.text.StringEscapeUtils; import java.text.Normalizer; @Route(\u0026#34;homoglyph-protection\u0026#34;) public class HomoglyphProtectionView extends VerticalLayout { public HomoglyphProtectionView() { // Username input field TextField userInputField = new TextField(\u0026#34;Enter username\u0026#34;); userInputField.setHelperText(\u0026#34;Only Latin letters and numbers allowed.\u0026#34;); // Validation button Button validateButton = new Button(\u0026#34;Validate\u0026#34;, event -\u0026gt; { String userInput = userInputField.getValue(); // Step 1: Unicode normalization String normalizedInput = Normalizer.normalize(userInput, Normalizer.Form.NFKC); // Step 2: Input validation - only Latin letters and numbers if (!isValidInput(normalizedInput)) { Notification.show(\u0026#34;Username contains invalid characters.\u0026#34;); return; } // Step 3: Escape potentially harmful input String escapedInput = StringEscapeUtils.escapeHtml4(normalizedInput); // Notification if input is valid Notification.show(\u0026#34;Input is valid: \u0026#34; + escapedInput); }); add(userInputField, validateButton); } // Input validation method that only allows Latin letters and numbers private boolean invalid input(String input) { return input.matches(\u0026#34;^[a-zA-Z0-9]*$\u0026#34;); } } In this Vaadin Flow application, what you have learned before is put into practice:\nUnicode normalisation : User input is normalised to ensure that visually similar characters that have different Unicode values ​​are treated correctly.\nInput validation : The input is checked to allow only Latin letters and numbers. This prevents the use of homoglyphs from other character sets.\nEscape and encode : The input is escaped before further processing to remove potentially harmful characters and prevent attacks such as Cross-Site Scripting (XSS).\nThis sample application shows how to apply the secure user input best practices discussed previously in a real-world web application to minimise homoglyph risks.\nConclusion # CWE-1007, insufficient visual discrimination of homoglyphs, is a vulnerability that is becoming increasingly important in today\u0026rsquo;s digital world. Attackers use the visual similarity of characters to deceive you, steal your data, or compromise systems. The threat of homoglyph attacks can have serious consequences, especially if you and your systems are not adequately trained and prepared.\nPreventing such attacks requires technical solutions, user training, and developer best practices. By implementing appropriate defence strategies, you can minimise the risk of homoglyph attacks and ensure the security of your systems and data. Everyone must be involved—from developers to end users—be aware of the risks and take steps to mitigate them.\nNext Steps # To better understand and protect yourself from CWE-1007, you should:\nTrain your employees about the danger of homoglyphs and teach them to recognize suspicious characters. Enable security features in your browsers and applications to warn users about fake domains. Integrate homoglyph recognition tools and libraries into your software development processes. Conduct regular security scans and code reviews to ensure no malicious characters have been inserted into your systems. Security is an ongoing process, and the homoglyph threat requires continued vigilance and adaptation. Stay informed, train your team, and use the right tools to ensure homoglyphs aren\u0026rsquo;t a gap in your security strategy. Together, we can help improve the security of our digital world and successfully ward off homoglyph threats.\nHappy Coding\nSven\n","date":"4 November 2024","externalUrl":null,"permalink":"/posts/what-is-cwe-1007-insufficient-visual-discrimination-of-homoglyphs-for-you-as-a-user/","section":"Posts","summary":"The world of cybersecurity is full of threats, many of which are surprisingly subtle and challenging to detect. One such threat is the problem of so-called homoglyphs. CWE-1007, also known as “Insufficient Visual Distinction of Homoglyphs Presented to User”, is a vulnerability often used by attackers to deceive and compromise your systems or data. In this blog article, you will get a deep insight into CWE-1007, understand its mechanisms, and how to protect yourself from such attacks. We will discuss examples, technical challenges, and best practices that can help you as a developer understand and mitigate this threat.\n","title":"What is CWE-1007: Insufficient visual discrimination of homoglyphs for you as a user?","type":"posts"},{"content":"","date":"1 November 2024","externalUrl":null,"permalink":"/tags/concurrency/","section":"Tags","summary":"","title":"Concurrency","type":"tags"},{"content":"","date":"1 November 2024","externalUrl":null,"permalink":"/tags/reentrantreadwritelock/","section":"Tags","summary":"","title":"ReentrantReadWriteLock","type":"tags"},{"content":"","date":"1 November 2024","externalUrl":null,"permalink":"/tags/semaphore/","section":"Tags","summary":"","title":"Semaphore","type":"tags"},{"content":"","date":"1 November 2024","externalUrl":null,"permalink":"/tags/stampedlock/","section":"Tags","summary":"","title":"StampedLock","type":"tags"},{"content":"Since the early days of computer science, parallel processing has represented one of the greatest challenges and opportunities. Since its inception in 1995, Java has undergone a significant journey in the world of parallel programming to provide developers with ever-better tools. This story is a fascinating journey through threads, the executor framework, fork/join, parallel streams, CompletableFuture and the latest developments in Project Loom. In this blog post, we take a detailed look at the evolution of parallel processing in Java and the innovations that came with it.\nLocking mechanisms in thread programming in Java synchronized keyword ReentrantLock ReadWriteLock StampedLock Semaphore Summary of the locking mechanisms The early years: Threads and Runnable Using threads: advantages and disadvantages Java 5: The Executor Framework and java.util.concurrent Java 7: Fork/Join Framework Java 8: Parallel Streams Java 8: CompletableFuture Java 9 to Java 19: Improvements and Project Loom The Evolution of Parallel Processing in Java: Conclusion Looking into the future Locking mechanisms in thread programming in Java # To make parallel processing safe and efficient, Java provides various locking mechanisms that help coordinate access to shared resources and avoid data corruption. These mechanisms are critical to ensure data consistency and integrity, especially when multiple threads access the same resources simultaneously. The most critical locking mechanisms in Java are described below:\nsynchronized keyword # The synchronized keyword is the most basic mechanism for synchronising threads in Java. It ensures that only one thread can access a synchronised method or code block at a time. This mechanism uses a so-called monitor, which is automatically locked and unlocked.\npublic synchronized void increment() { // Critical section counter++; } Advantages:\n- Easy to use and integrate into code.\n- Automatic management of bans.\nDisadvantages:\n- Can lead to performance issues if many threads want to access the synchronised resource simultaneously.\n- No flexibility, such as B. the ability to set timeouts or conditional locks.\nReentrantLock # ReentrantLock is a class from the java.util.concurrent.locks package that provides greater flexibility than the synchronized keyword. As the name suggests, ReentrantLock is a \u0026ldquo;reentrant\u0026rdquo; lock, meaning that a thread that already holds the lock can reacquire it without getting into a deadlock.\nprivate final ReentrantLock lock = new ReentrantLock(); public void increment() { lock.lock(); try { // Critical section counter++; } finally { lock.unlock(); } } Advantages:\n- Provides more control over the locking process, e.g. B. the ability to acquire locks with timeouts (tryLock).\n- Supports fair locking, which ensures that threads are granted access in the requested order (lock(true)).\nDisadvantages:\n- Requires manual release of locks, increasing the risk of deadlocks if unlock is not performed correctly.\nReadWriteLock # ReadWriteLock is a unique locking mechanism that distinguishes read and write locks. It consists of two locks: a read lock and a write lock. Multiple threads can acquire a read lock simultaneously as long as no write operation is performed. However, only one thread can acquire the write lock, keeping writes exclusive.\nprivate final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(); public void read() { lock.readLock().lock(); try { // Read operation } finally { lock.readLock().unlock(); } } public void write() { lock.writeLock().lock(); try { // Write operation } finally { lock.writeLock().unlock(); } } Advantages:\n- Increases concurrency as multiple threads can read simultaneously as long as there are no writes.\n- Reduces the likelihood of blocking read access.\nDisadvantages:\n- More complex to use than synchronized or ReentrantLock.\n- Requires careful design to ensure no deadlocks or race conditions occur.\nStampedLock # StampedLock is another variant of ReadWriteLock introduced in Java 8. Unlike ReadWriteLock, StampedLock provides optimistic read locks that enable even higher concurrency. A stamp is returned each time the lock is acquired to ensure the data remains valid.\nprivate final StampedLock lock = new StampedLock(); public void read() { long stamp = lock.tryOptimisticRead(); // Read operation if (!lock.validate(stamp)) { stamp = lock.readLock(); try { // Read again } finally { lock.unlockRead(stamp); } } } Advantages:\n- Provides optimistic read locks that enable high concurrency as long as no writes occur.\n- Lower read operation overhead compared to traditional locks.\nDisadvantages:\n- Complex to use as developers need to ensure that the stamp is valid.\n- No “re-enterable” locks, which may limit usage for some scenarios.\nSemaphore # Semaphore is another synchronisation mechanism that allows several threads to access a resource simultaneously. It is often used to control simultaneous access to limited resources.\nprivate final Semaphore semaphore = new Semaphore(3); public void accessResource() { try { semaphore.acquire(); // Access the resource } catch (InterruptedException e) { Thread.currentThread().interrupt(); } finally { semaphore.release(); } } Advantages:\n- Allows you to limit the number of threads that are allowed to access a resource at the same time.\n- Flexible and applicable to various scenarios, e.g. B. to implement pooling mechanisms.\nDisadvantages:\n- Can become complex when using multiple semaphores in an application.\n- Requires careful management to ensure that acquire and release are called correctly.\nSummary of the locking mechanisms # Java offers a variety of locking mechanisms, each suitable for different scenarios. The synchronized keyword is easy to use, but less flexible for complex scenarios. ReentrantLock and ReadWriteLock provide more control and enable higher parallelism, while StampedLock is suitable for particularly demanding read operations. Semaphore, on the other hand, is ideal for controlling concurrent access to limited resources. Choosing the proper mechanism depends on the application\u0026rsquo;s requirements, particularly regarding concurrency, resource contention, and maintainability.\nThe early years: Threads and Runnable # Initially, Java\u0026rsquo;s concept of threads was the cornerstone of parallel processing. Java was developed as a platform-independent language intended for networked, distributed systems. With a focus on multithreading, Java offered a built-in Thread class and the Runnable interface, making it possible to execute multiple tasks simultaneously.\nclass MyTask implements Runnable { @Override public void run() { // Task running in parallel System.out.println(\u0026#34;Hello from thread: \u0026#34; + Thread.currentThread().getName()); } } public class Main { public static void main(String[] args) { Thread thread = new Thread(new MyTask()); thread.start(); } } In the early versions of Java, threads were the only option for parallel processing. These were supported by operating system threads, which meant managing threads could be heavy and resource-intensive. Developers had to take care of details such as synchronizing shared resources and avoiding deadlocks themselves, which made developing parallel applications challenging.\nUsing threads: advantages and disadvantages # Advantages of threads\nSimple modelling of parallel tasks : Threads allow developers to divide parallel tasks into separate units that can run concurrently.\nDirect control : Threads give developers fine control over parallel execution, which is particularly useful when specific thread management requirements exist.\nGood support from the operating system : Threads are supported directly by the operating system, meaning they can access all system resources and benefit from operating system optimisations.\nDisadvantages of threads\nComplexity of synchronisation : When multiple threads access shared resources, developers must use synchronisation mechanisms such as synchronized blocks or locks to avoid data corruption. Such mechanisms ensure that only one thread can access a critical resource at a time, ensuring data consistency. However, this often results in complicated and error-prone code, as developers must carefully ensure that all necessary sections are correctly synchronised. An incorrectly set or forgotten synchronisation block can lead to severe errors, such as race conditions, in which the program\u0026rsquo;s output depends on the timing of thread executions. Additionally, synchronisation mechanisms such as locks or synchronized blocks can lead to a performance penalty as threads often have to wait until a resource is released, which limits parallelism. These wait times can cause bottlenecks in more complex applications, mainly when multiple threads compete for different resources. Therefore, correctly applying synchronisation techniques requires a deep understanding of thread interactions and careful design to ensure data integrity and maximise performance.\nResource intensive : Each thread requires space for its stack and additional resources such as thread handling and context switching. These resource requirements can quickly add up with many threads and lead to system overload. In particular, the memory consumption for the thread stacks and the management of the threads by the operating system lead to increased resource requirements. With a large number of threads, the frequency of context switches also increases, which can lead to a significant reduction in overall performance. This often makes threads inefficient and difficult to manage at a large scale.\nThe danger of deadlocks : When using synchronisation mechanisms, there is a risk of deadlocks when two or more threads are blocked in a cyclic dependency on each other. Deadlocks often arise when multiple threads hold different locks in a mismatched order, waiting for other threads to release the needed resources. This leads to a situation where none of the threads can continue working because they are all waiting for the others to release resources. Deadlocks are often difficult to reproduce and debug because they only occur under certain runtime conditions. Strategies to avoid deadlocks include using timeout mechanisms, avoiding nested locks, and implementing lock ordering schemes to ensure all threads acquire locks in the same order.\nDifficult scalability : Manual management of threads makes it difficult to scale applications, especially on systems with many processor cores. One of the main reasons for this is the challenge of determining the optimal number of threads to use system resources efficiently. Too many threads can cause system overload because management and context switching between threads consume significant CPU resources. On the other hand, too few threads can result in under-utilization of available processor resources, degrading overall performance. Additionally, it is challenging to adapt thread management to the dynamic needs of an application, especially when the load is variable. Developers often have to resort to complex heuristics or dynamic thread pools to control the number of active threads, which significantly complicates the implementation and maintenance of the application. These challenges make it complicated to efficiently scale applications to modern multicore processors because the balance between parallelism and overhead is challenging.\nJava 5: The Executor Framework and java.util.concurrent # The introduction of Java 5 in 2004 introduced the java.util.concurrent package, intended to address many of the problems of early parallel programming in Java. The Executor framework enabled higher abstraction of threads. Instead of starting and managing threads manually, developers could now rely on a task-based architecture to pass tasks to an ExecutorService.\nThe Executor framework introduced classes like ThreadPoolExecutor, ScheduledExecutorService, and many synchronization helper classes like Semaphore, CountDownLatch, and ConcurrentHashMap. This not only made thread management easier but also led to more efficient use of system resources.\nExecutorService executor = Executors.newFixedThreadPool(4); executor.submit(() -\u0026gt; { System.out.println(\u0026#34;Task is running in thread: \u0026#34; + Thread.currentThread().getName()); }); executor.shutdown(); The Executor Framework changed the way developers modelled parallel tasks. Instead of focusing on thread creation, they could define tasks and let the infrastructure handle execution.\nJava 7: Fork/Join Framework # Java 7 introduced the Fork/Join framework in 2011, explicitly designed for computationally intensive tasks that could be broken down into smaller subtasks. The fork/join framework provided a powerful recursion and divide-and-conquer infrastructure, allowing complex problems to be broken down into smaller, more manageable sub-problems. This division enabled the efficient use of modern multi-core processors.\nThe fork/join framework was particularly useful for problems broken down into independent subproblems, such as B. calculating Fibonacci numbers, sorting large arrays (e.g. with merge sort) or processing large amounts of data in parallel. The central component of the framework is the ForkJoinPool, which handles the management of task transfer between threads. The ForkJoinPool uses the so-called work stealing process, in which less busy threads take over work from more busy threads. This ensures better load balancing and increases the efficiency of parallel processing.\nAnother advantage of the fork/join framework is the ability to handle recursive tasks efficiently. Developers can use classes like RecursiveTask or RecursiveAction to define tasks that either provide a return value (RecursiveTask) or do not require a return (RecursiveAction). The fork/join approach makes it possible to recursively split the tasks (fork) and then combine the results again (join).\nForkJoinPool forkJoinPool = new ForkJoinPool(); forkJoinPool.invoke(new RecursiveTask\u0026lt;Integer\u0026gt;() { @Override protected Integer compute() { // Split and calculate task if (/* base case reached */) { return 42; // Example return value } else { // Divide the task further RecursiveTask\u0026lt;Integer\u0026gt; subtask1 = new Subtask(); RecursiveTask\u0026lt;Integer\u0026gt; subtask2 = new Subtask(); subtask1.fork(); subtask2.fork(); return subtask1.join() + subtask2.join(); } } }); The fork/join framework delivered significant performance improvements for CPU-intensive workloads, particularly for tasks easily broken down into smaller, independent pieces. It made it easier to write parallel algorithms without the developer worrying about thread distribution details. The \u0026lsquo;ForkJoinPool\u0026rsquo; handles the distribution of tasks and uses work stealing to ensure that the processor resources are used optimally. This significantly increases performance compared to manual thread management, especially for computationally intensive and highly parallelisable tasks.\nJava 8: Parallel Streams # Java 8, released in 2014, marked a milestone in the evolution of Java, particularly with the introduction of lambda expressions, streams, and functional programming. These new features made the language more flexible and easier to use, especially for parallel operations. One of the most significant new features for parallel processing was Parallel Streams.\nParallel Streams allowed developers to effortlessly parallelise operations across collections without explicitly dealing with threads or synchronisation. This was achieved by integrating the fork/join framework behind the scenes. Parallel streams use the ForkJoinPool internally to distribute tasks efficiently across multiple processor cores. This approach is based on the divide-and-conquer design principle, in which an enormous task is broken down into smaller subtasks that can be executed in parallel.\nThe developer uses the parallelStream() method to convert a collection into a parallel stream. This results in the processing of the collection\u0026rsquo;s elements occurring simultaneously, with individual parts of the task being automatically distributed across the available CPU cores. In contrast to manual thread management, this approach offers a high level of abstraction and relieves the developer of the complex management of threads and synchronisation.\nAn example of using Parallel Streams is processing a list of numbers in parallel. Here, the operations applied to each element are carried out simultaneously, which can achieve a significant increase in performance on large data sets:\nList\u0026lt;Integer\u0026gt; numbers = Arrays.asList(1, 2, 3, 4, 5); numbers.parallelStream().map(n -\u0026gt; n * n).forEach(System.out::println); The design of Parallel Streams aims to simplify the development of parallel applications by using a declarative syntax that allows the developer to focus on the logic of data processing rather than on the details of thread management and distribution of the tasks to deal with. This higher abstraction makes parallel processing more accessible, which leads to better performance, especially in multi-core systems.\nJava 8: CompletableFuture # The CompletableFuture API was also introduced in Java 8 and significantly expanded the possibilities of asynchronous programming. CompletableFuture allows the creation, chaining and combining asynchronous tasks, making it a handy tool for developing event-driven applications. Using methods like thenApply, thenAccept and thenCombine makes it easy to define a sequence of asynchronous operations that should be executed sequentially or whose results can be combined.\nA CompletableFuture represents a future calculation and allows the different steps of the workflow to be defined declaratively. For example, an asynchronous calculation can be started, and the result can be processed further without worrying about explicitly managing the threads. This significantly simplifies the code and makes it more readable.\nAn example of using CompletableFuture shows how to execute multiple asynchronous operations one after the other:\nCompletableFuture.supplyAsync(() -\u0026gt; \u0026#34;Hello\u0026#34;) .thenApply(s -\u0026gt; s + \u0026#34; World\u0026#34;) .thenAccept(System.out::println); In this example, an asynchronous calculation is first started that returns the string \u0026ldquo;Hello\u0026rdquo;. Then, the result of this calculation is modified by the thenApply method by adding \u0026quot; World\u0026quot;. Finally, thenAccept prints the result on the console. The entire process occurs asynchronously, without the developer worrying about explicitly managing threads.\nThe architecture of CompletableFuture is based on the concept of so-called “completion stages”, which allow the creation of asynchronous pipelines. Each Completion Stage can either trigger a new calculation or process the result of a previous calculation. This enables the modelling of complex workflows, such as: E.g. executing multiple tasks in parallel and then combining the results (thenCombine), or defining actions that should be carried out in the event of an error (exceptionally).\nAnother significant advantage of CompletableFuture is the ability to combine asynchronous tasks. For example, two independent asynchronous calculations can be performed in parallel, and their results can then be merged:\nCompletableFuture\u0026lt;Integer\u0026gt; future1 = CompletableFuture.supplyAsync(() -\u0026gt; 10); CompletableFuture\u0026lt;Integer\u0026gt; future2 = CompletableFuture.supplyAsync(() -\u0026gt; 20); CompletableFuture\u0026lt;Integer\u0026gt; combinedFuture = future1.thenCombine(future2, (result1, result2) -\u0026gt; result1 + result2); combinedFuture.thenAccept(result -\u0026gt; System.out.println(\u0026#34;Combined Result: \u0026#34; + result)); In this example, two calculations are performed in parallel, and their results are combined. The thenCombine method allows the results of the two futures to be added, and thenAccept prints the combined result.\nThis model allows complex workflows to be created without explicitly relying on threads or callbacks, making the code cleaner, more modular, and easier to maintain. CompletableFuture also provides methods such as allOf() and anyOf() that allow multiple futures to be monitored and processed simultaneously. This is particularly useful for scenarios where numerous independent tasks must be executed in parallel.\nOverall, the CompletableFuture API makes asynchronous programming in Java much more accessible and allows developers to develop reactive and non-blocking applications with relatively little effort.\nJava 9 to Java 19: Improvements and Project Loom # After Java 8, the parallel programming models continuously improved. Java 9 brought improvements to the fork/join framework and introduced the \u0026lsquo;Flow\u0026rsquo; API, which supported a reactive streaming model. Java 9 to 17 focused primarily on performance improvements, security, and the modularization of the JDK (Project Jigsaw).\nHowever, one of the most significant innovations in recent times is Project Loom. Since Java 19, Virtual Threads has been available as a preview feature to revolutionise parallel programming in Java. Virtual threads are lightweight threads that enable many concurrent tasks to run without the typical overhead of traditional operating system threads. While traditional threads are limited by resource costs (such as memory for stacks and context switches), virtual threads are designed to be much more resource-efficient. This means developers can create millions of virtual threads that work independently without overloading the system.\nVirtual threads are handy for server-side applications that handle many concurrent connections, such as web servers or microservices. In traditional approaches, handling each request in its thread often leads to scaling problems because hardware resources limit the number of possible threads. Virtual threads, on the other hand, allow each incoming request to be handled in its virtual thread, significantly increasing parallelism.\nVirtual threads work because they are efficiently managed by the Java runtime system rather than directly by the operating system like traditional threads. This significantly speeds up context switching and makes managing millions of threads realistic. Virtual threads are programmed like traditional threads, allowing existing code to be easily adapted to take advantage of this new technology.\nA simple example of using virtual threads shows how to use an executor to execute a task in a virtual thread:\ntry (var executor = Executors.newVirtualThreadPerTaskExecutor()) { executor.submit(() -\u0026gt; { System.out.println(\u0026#34;Running in a virtual thread\u0026#34;); }); } This example creates an executor that uses a new virtual thread for each task. The submit method starts the task in a virtual thread, which requires significantly fewer resources than a traditional thread.\nProject Loom can potentially make parallel programming in Java much more accessible by eliminating the need for developers to worry about thread scaling. Virtual threads are significantly more efficient and offer much higher parallelism without the programmer having to work with thread pools or complex synchronisation mechanisms explicitly. This increased scalability is particularly valuable in applications where concurrent operations must be dynamically scaled, as in many modern web applications and cloud environments. The introduction of virtual threads makes Java an even stronger choice for developing highly scalable, parallel applications by dramatically reducing the complexity of thread management.\nThe Evolution of Parallel Processing in Java: Conclusion # The journey of parallel processing in Java reflects the language\u0026rsquo;s evolutionary nature. From the early days of threads, when developers had to rely on low-level APIs, to the highly abstract paradigms such as the Executor framework, Fork/Join, and Parallel Streams, Java has continually introduced improvements to make parallel application development easier.\nWith recent developments such as \u0026lsquo;CompletableFuture\u0026rsquo; and Project Loom, Java can meet the needs of modern software development, especially in scalability and performance. Parallel processing in Java is now simpler, safer and more efficient than ever before - providing developers with powerful tools to exploit the full potential of modern multicore systems.\nLooking into the future # With Project Loom on the path to stability, we could see a further shift in focus towards even simpler and more performant parallel processing techniques. Virtual threads will likely pave the way for new frameworks and libraries that benefit from this lightweight parallelism. Developers will continue to have access to the best parallel processing tools without worrying about thread management\u0026rsquo;s complexities.\nJava has proven that it can keep changing with the times as a parallel programming language—a language that rises to the challenges and adapts to the needs of modern developers. Java\u0026rsquo;s history of parallel processing is one of progress and continuous innovation. With the upcoming developments, it will remain exciting to see what new possibilities the future will bring us.\n","date":"1 November 2024","externalUrl":null,"permalink":"/posts/the-history-of-parallel-processing-in-java-from-threads-to-virtual-threads/","section":"Posts","summary":"Since the early days of computer science, parallel processing has represented one of the greatest challenges and opportunities. Since its inception in 1995, Java has undergone a significant journey in the world of parallel programming to provide developers with ever-better tools. This story is a fascinating journey through threads, the executor framework, fork/join, parallel streams, CompletableFuture and the latest developments in Project Loom. In this blog post, we take a detailed look at the evolution of parallel processing in Java and the innovations that came with it.\n","title":"The History of Parallel Processing in Java: From Threads to Virtual Threads","type":"posts"},{"content":"","date":"18 October 2024","externalUrl":null,"permalink":"/tags/cwe-778/","section":"Tags","summary":"","title":"CWE-778","type":"tags"},{"content":" Learn how inadequate control over error reporting leads to security vulnerabilities and how to prevent them in Java applications. # Safely handling error reports is a central aspect of software development, especially in safety-critical applications. CWE-778 describes a vulnerability caused by inadequate control over error reports. This post will analyse the risks associated with CWE-778 and show how developers can implement safe error-handling practices to avoid such vulnerabilities in Java programs.\nLearn how inadequate control over error reporting leads to security vulnerabilities and how to prevent them in Java applications. What is CWE-778? Examples of CWE-778 in Java Secure error handling Example with Vaadin Flow Using design patterns to reuse logging and error handling. Decorator Pattern Proxy Pattern Template Method Pattern Evaluate log messages for attack detection in real time Best practices for avoiding CWE-778 Conclusion What is CWE-778? # The Common Weakness Enumeration (CWE) defines CWE-778 as a vulnerability where bug reporting is inadequately controlled. Bug reports often contain valuable information about an application\u0026rsquo;s internal state, including system paths, configuration details, and other sensitive information that attackers can use to identify and exploit vulnerabilities. Improper handling of error reports can result in unauthorised users gaining valuable insight into the application\u0026rsquo;s system structure and logic.\nExposing such information in a security-sensitive application could have potentially serious consequences, such as the abuse of SQL injection or cross-site scripting (XSS) vulnerabilities. Therefore, it is critical that bug reports are carefully controlled and only accessible to authorised individuals.\nExamples of CWE-778 in Java # The following example considers a simple Java application used to authenticate users:\npublic class UserLogin { public static void main(String[] args) { try { authenticateUser(\u0026#34;admin\u0026#34;, \u0026#34;wrongpassword\u0026#34;); } catch (Exception e) { // Error is output directly to the user System.out.println(\u0026#34;Error: \u0026#34; + e.getMessage()); e.printStackTrace(); } } private static void authenticateUser(String username, String password) throws Exception { if (!\u0026#34;correctpassword\u0026#34;.equals(password)) { throw new Exception(\u0026#34;Invalid password for user: \u0026#34; + username); } } } This example displays an error message if the user enters an incorrect password. However, this approach has serious security gaps:\n1. The error message contains specific information about the username.\n2. The full stack trace is output, allowing an attacker to obtain details about the application\u0026rsquo;s implementation.\nThis information can help an attacker understand the application\u0026rsquo;s internal structure and make it easier for them to search specifically for additional vulnerabilities.\nSecure error handling # To minimise the risks described above, secure error handling should be implemented. Instead of providing detailed information about the error, the user should only be shown a general error message:\npublic class UserLogin { public static void main(String[] args) { try { authenticateUser(\u0026#34;admin\u0026#34;, \u0026#34;wrongpassword\u0026#34;); } catch (Exception e) { // Generic error message to the user System.out.println(\u0026#34;Authentication failed. Please check your entries.\u0026#34;); // Logging the error in the log file (for admins) logError(e); } } private static void authenticateUser(String username, String password) throws Exception { if (!\u0026#34;correctpassword\u0026#34;.equals(password)) { throw new Exception(\u0026#34;Invalid password for user: \u0026#34; + username); } } private static void logError(Exception e) { // Error is securely logged without displaying it to the user System.err.println(\u0026#34;An error has occurred: \u0026#34; + e.getMessage()); } } In this improved version, only a general error message is displayed to the user while the error is logged internally. This prevents sensitive information from being shared with unauthorised users.\nSuch errors should be logged in a log file accessible only to authorised persons. A logging framework such as Log4j or SLF4J provides additional mechanisms to ensure logging security and store only necessary information.\nExample with Vaadin Flow # Vaadin Flow is a Java framework for building modern web applications, and CWE-778 can also be a problem if error reports are mishandled. A safe example of error handling in a Vaadin application could look like this:\nimport com.vaadin.flow.component.button.Button; import com.vaadin.flow.component.notification.Notification; import com.vaadin.flow.component.textfield.PasswordField; import com.vaadin.flow.component.textfield.TextField; import com.vaadin.flow.router.Route; @Route(\u0026#34;login\u0026#34;) public class LoginView extends VerticalLayout { public LoginView() { TextField usernameField = new TextField(\u0026#34;Benutzername\u0026#34;); PasswordField passwordField = new PasswordField(\u0026#34;Password\u0026#34;); Button loginButton = new Button(\u0026#34;Login\u0026#34;, event -\u0026gt; { try { authenticateUser(usernameField.getValue(), passwordField.getValue()); } catch (Exception e) { // Generic error message to the user Notification.show(\u0026#34;Authentication failed. Please check your entries.\u0026#34;); // Logging the error in the log file (for admins) logError(e); } }); add(usernameField, passwordField, loginButton); } private void authenticateUser(String username, String password) throws Exception { if (!\u0026#34;correctpassword\u0026#34;.equals(password)) { throw new Exception(\u0026#34;Invalid password for user: \u0026#34; + username); } } private void logError(Exception e) { // Error is securely logged without displaying it to the user System.err.println(\u0026#34;An error has occurred: \u0026#34; + e.getMessage()); } } The logError method ensures that errors are logged securely without sensitive information being visible to the end user. Vaadin Flow enables the integration of such secure practices to ensure that bug reports are not leaked uncontrollably.\nUsing design patterns to reuse logging and error handling. # To promote the reuse of error handling and logging, design patterns that enable the modularization and unification of such tasks can be used. Two suitable patterns are the Decorator Pattern and the Template Method Pattern.\nDecorator Pattern # The Decorator Pattern is a structural design pattern that allows an object\u0026rsquo;s functionality to be dynamically extended without changing the underlying class. This is particularly useful when adding additional responsibilities, such as logging, security checks, or error handling, without modifying the original class\u0026rsquo;s code.\nThe Decorator Pattern works by using so-called “wrappers”. Instead of modifying the class directly, the object is wrapped in another class that implements the same interface and adds additional functionality. In this way, different decorators can be combined to create a flexible and expandable structure.\nA vital feature of the Decorator Pattern is its adherence to the open-closed principle, one of the fundamental principles of object-oriented design. The open-closed principle states that a software component should be open to extensions but closed to modifications. The Decorator Pattern does just that by allowing classes to gain new functionality without changing their source code.\nIn the context of error handling and logging, developers can write an introductory class for authentication, while separate decorators handle error logging and handling of specific errors. This leads to a clear separation of responsibilities, significantly improving code maintainability.\nThe following example shows the implementation of the Decorator pattern to reuse error handling and logging:\npublic interface Authenticator { void authenticate(String username, String password) throws Exception; } public class BasicAuthenticator implements Authenticator { @Override public void authenticate(String username, String password) throws Exception { if (!\u0026#34;correctpassword\u0026#34;.equals(password)) { throw new Exception(\u0026#34;Invalid password for user: \u0026#34; + username); } } } public class LoggingAuthenticatorDecorator implements Authenticator { private final Authenticator wrapped; public LoggingAuthenticatorDecorator(Authenticator wrapped) { this.wrapped = wrapped; } @Override public void authenticate(String username, String password) throws Exception { try { wrapped.authenticate(username, password); } catch (Exception e) { logError(e); throw e; } } private void logError(Exception e) { System.err.println(\u0026#34;An error has occurred: \u0026#34; + e.getMessage()); } } In this example, **BasicAuthenticator** is used as the primary authentication class, while the **LoggingAuthenticatorDecorator** Added additional functionality, namely error logging. This decorator wraps the original authentication class and extends its behaviour. This allows the logic to be flexibly extended by adding more decorators, such as a **SecurityCheckDecorator** , which performs additional security checks before authentication.\nAn advantage of this approach is combining decorators in any order to achieve tailored functionality. For example, one could first add a security decoration and then implement error logging without changing the original authentication logic. This results in a flexible and reusable structure that is particularly useful in large projects where different aspects such as logging, security checks, and error handling are required in various combinations.\nThe Decorator Pattern is, therefore, a powerful tool for increasing software modularity and extensibility. It avoids code duplication, promotes reusability, and enables a clean separation of core logic and additional functionalities. This makes it particularly useful in secure error handling and implementing cross-cutting concerns such as logging in safety-critical applications.\nThe Decorator Pattern can add functionality, such as logging or error handling, to existing methods without modifying their original code. The following example shows how the Decorator Pattern enables centralised error handling:\npublic interface Authenticator { void authenticate(String username, String password) throws Exception; } public class BasicAuthenticator implements Authenticator { @Override public void authenticate(String username, String password) throws Exception { if (!\u0026#34;correctpassword\u0026#34;.equals(password)) { throw new Exception(\u0026#34;Invalid password for user: \u0026#34; + username); } } } public class LoggingAuthenticatorDecorator implements Authenticator { private final Authenticator wrapped; public LoggingAuthenticatorDecorator(Authenticator wrapped) { this.wrapped = wrapped; } @Override public void authenticate(String username, String password) throws Exception { try { wrapped.authenticate(username, password); } catch (Exception e) { logError(e); throw e; } } private void logError(Exception e) { System.err.println(\u0026#34;An error has occurred: \u0026#34; + e.getMessage()); } } In this example, the **LoggingAuthenticatorDecorator** is a decorator for the class **BasicAuthenticator** . The Decorator Pattern allows error handling and logging to be centralised without changing the underlying authentication class.\nProxy Pattern # The proxy pattern is a structural design pattern used to control access to an object. It is often used to add functionality such as caching, access control, or logging. In contrast to the decorator pattern, primarily used to extend functionality, the proxy serves as a proxy that takes control of access to the actual object.\nThe proxy pattern ensures that all access to the original object occurs via the proxy, meaning specific actions can be carried out automatically. For example, the Proxy Pattern could ensure that authorised users can only access a particular resource while logging all accesses.\nA typical example of the proxy pattern for encapsulating logging and error handling looks like this:\npublic interface Authenticator { void authenticate(String username, String password) throws Exception; } public class BasicAuthenticator implements Authenticator { @Override public void authenticate(String username, String password) throws Exception { if (!\u0026#34;correctpassword\u0026#34;.equals(password)) { throw new Exception(\u0026#34;Invalid password for user: \u0026#34; + username); } } } public class ProxyAuthenticator implements Authenticator { private final Authenticator realAuthenticator; public ProxyAuthenticator(Authenticator realAuthenticator) { this.realAuthenticator = realAuthenticator; } @Override public void authenticate(String username, String password) throws Exception { logAccessAttempt(username); try { realAuthenticator.authenticate(username, password); } catch (Exception e) { logError(e); throw e; } } private void logAccessAttempt(String username) { System.out.println(\u0026#34;Authentication attempt for user: \u0026#34; + username); } private void logError(Exception e) { System.err.println(\u0026#34;An error has occurred: \u0026#34; + e.getMessage()); } } In this example, BasicAuthenticator is wrapped by a **ProxyAuthenticator** , which controls all calls to the authenticate method. The proxy adds additional functionality, such as access and error logging, ensuring that all access goes through the proxy before the authentication object is called.\nA key difference between the Proxy Pattern and the Decorator Pattern is that the Proxy primarily controls access to the object and its use. The proxy can check access rights, add caching, or manage an object\u0026rsquo;s lifetime. The Decorator Pattern, on the other hand, is designed to extend an object\u0026rsquo;s behaviour by adding additional responsibilities without changing the access logic.\nIn other words, the Proxy Pattern acts as a protection or control mechanism, while the Decorator Pattern adds additional functionality to extend the behaviour. Both patterns are very useful when integrating cross-cutting concerns such as logging or security checks into the application, but they differ in their focus and application.\nTemplate Method Pattern # The Template Method Pattern allows for defining the general flow of a process while implementing specific steps in subclasses. This ensures that error handling remains consistent:\npublic abstract class AbstractAuthenticator { public final void authenticate(String username, String password) { try { doAuthenticate(username, password); } catch (Exception e) { logError(e); throw new RuntimeException(\u0026#34;Authentication failed. Please check your entries.\u0026#34;); } } protected abstract void doAuthenticate(String username, String password) throws Exception; private void logError(Exception e) { System.err.println(\u0026#34;An error has occurred: \u0026#34; + e.getMessage()); } } public class ConcreteAuthenticator extends AbstractAuthenticator { @Override protected void doAuthenticate(String username, String password) throws Exception { if (!\u0026#34;correctpassword\u0026#34;.equals(password)) { throw new Exception(\u0026#34;Invalid password for user: \u0026#34; + username); } } } The Template Method Pattern centralises error handling in the **AbstractAuthenticator** class so that all subclasses use the same consistent error handling strategy.\nEvaluate log messages for attack detection in real time # Another aspect of secure error handling is using log messages to detect attacks in real-time. Analysing the log data can identify potential attacks early, and appropriate measures can be taken. The following approaches are helpful:\nCentralised logging : Use a central logging platform like the ELK Stack (Elasticsearch, Logstash, Kibana) or Splunk to collect all log data in one place. This enables comprehensive analysis and monitoring of security-related incidents.\nPattern recognition : Create rules and patterns that identify potentially malicious activity, such as multiple failed login attempts in a short period. Such rules can trigger automated alerts when suspicious activity is detected.\nAnomaly detection : Machine learning techniques detect anomalous activity in log data. A sudden increase in certain error messages or unusual access patterns could indicate an ongoing attack.\nReal-time alerts : Configure the system so that certain security-related events trigger alerts in real-time. This allows administrators to respond immediately to potential threats.\nAnalyse threat intelligence : Use log messages to collect and analyse threat intelligence. For example, IP addresses that repeatedly engage in suspicious activity can be identified, and appropriate action can be taken, such as blocking the address.\nIntegration into SIEM systems : Use security information and event management (SIEM) systems to correlate log data from different sources and gain deeper insights into potential threats. SIEM systems often also provide tools to automate responses to specific events.\nBy combining these approaches, attacks can be detected early, and the necessary steps can be taken to limit the damage.\nBest practices for avoiding CWE-778 # To avoid CWE-778 in your applications, the following best practices should be followed:\nGeneric error messages : Avoid sharing detailed information about errors with end users. Error messages should be worded as generally as possible to avoid providing clues about the internal implementation.\nError logging : Use logging frameworks like Log4j or SLF4J to log errors securely. This allows bugs to be tracked internally without exposing sensitive information.\nNo stack traces to users : Make sure stack traces are only visible in the log and are not output to the user. Instead, generic error messages that do not contain technical details should be used.\nAccess control : Ensure that only authorized users have access to detailed error reports. Error logs should be well-secured and viewable only by administrators or developers.\nRegular error testing and security analysis : Run regular tests to ensure that error handling works correctly. Static code analysis tools help detect vulnerabilities like CWE-778 early.\nAvoiding sensitive information : To prevent sensitive information such as usernames, passwords, file paths, or server details from being included in error messages, such information should only be stored in secured log files.\nUsing secure libraries : Rely on proven libraries and frameworks for error handling and logging that have already undergone security checks. This reduces the likelihood of implementation errors compromising security.\nConclusion # CWE-778 poses a severe security threat if bug reports are not adequately controlled. Developers must know the importance of handling errors securely to prevent unwanted information leaks. Applying secure programming practices, such as using design patterns to reuse error-handling logic and implementing centralised logging to detect attacks in real-time, can significantly increase the security and robustness of Java applications.\nSecure error handling improves an application\u0026rsquo;s robustness and user experience by providing clear and useful instructions without overwhelming the user with technical details. The combination of security and usability is essential for the success and security of modern applications.\nUltimately, control over bug reports is integral to a software project\u0026rsquo;s overall security strategy. Bug reports can either be a valuable resource for developers or, if handled poorly, become a vulnerability for attackers to exploit. Disciplined error handling, modern design patterns, and attack detection technologies are critical to ensuring that error reports are used as a tool for improvement rather than a vulnerability.\n","date":"18 October 2024","externalUrl":null,"permalink":"/posts/cwe-778-lack-of-control-over-error-reporting-in-java/","section":"Posts","summary":"Learn how inadequate control over error reporting leads to security vulnerabilities and how to prevent them in Java applications. # Safely handling error reports is a central aspect of software development, especially in safety-critical applications. CWE-778 describes a vulnerability caused by inadequate control over error reports. This post will analyse the risks associated with CWE-778 and show how developers can implement safe error-handling practices to avoid such vulnerabilities in Java programs.\n","title":"CWE-778: Lack of control over error reporting in Java","type":"posts"},{"content":"Unit testing is an essential software development concept that improves code quality by ensuring that individual units or components of a software function correctly. Unit testing is crucial in Java, one of the most commonly used programming languages. This article will discuss what unit testing is, how it has evolved, and what tools and best practices have been established over the years.\nDefinition von Unit Testing The history of unit testing The 1990s and the emergence of JUnit The 2000s: Agile Methods and Test-Driven Development (TDD) The 2010s: Integration into CI/CD pipelines and the importance of automation Tools and frameworks for unit testing in Java Best practices for unit testing in Java The future of unit testing What are the disadvantages of unit testing? Time expenditure and costs Limited test coverage False sense of security Excessive mocking Difficulties in testability of legacy code Not suitable for complex test cases Test-driven development (TDD) can lead to excessive focus on details Test maintenance for rapid changes Lack of testing strategies in complex systems Additional effort without immediate benefit Conclusion Does Secure Pay Load Testing belong to the area of ​​unit testing? What is Secure Payload Testing? Difference between Unit Testing and Secure Payload Testing Where does Secure Payload Testing fit in? Can Secure Payload Testing be part of Unit Testing? Which secure coding practices play a role in connection with unit testing? Input validation and sanitisation Limit value analysis (boundary testing) Secure error handling Avoiding hard-coded secrets Use of safe libraries and dependencies Ensure encryption Least Privilege Principle Avoiding race conditions Avoidance of buffer overflows Safe use of third-party libraries Definition von Unit Testing # Unit Testing refers to testing a program\u0026rsquo;s most minor functional units—usually individual functions or methods. A unit test ensures that a specific code function works as expected and typically checks whether a method or class returns the correct output for particular inputs.\nIn Java, unit testing often refers to testing individual methods in a class, testing whether the code responds as expected to both average and exceptional input.\nExample in Java:\npublic class Calculator { public int add(int a, int b) { return a + b; } } A unit test for the add method could look like this:\npublic class CalculatorTest { @Test public void testAdd() { Calculator calc = new Calculator(); assertEquals(5, calc.add(2, 3)); } } This example tests whether the add method correctly combines two numbers. If the test is successful, the process works as expected.\nThe history of unit testing # The roots of unit testing lie in the early days of software development. As early as the 1960s, programmers recognised that it was essential to ensure that individual parts of a program were working correctly before integrating the entire system. However, unit testing only became primarily standardised in the 1990s with the spread of object-oriented programming languages ​​such as Java and C++.\nThe 1990s and the emergence of JUnit # One of the most influential events in the history of unit testing was the introduction of the framework JUnit for the Java programming language in 1999. Developed by Kent Beck and Erich Gamma, JUnit revolutionised how Java developers test their software.\nJUnit enabled developers to write simple and easy-to-maintain unit tests for their Java programs. Before JUnit, no widely used, standardised tools facilitated unit testing in Java. Programmers had to create their test suites manually, which was tedious and error-prone.\nJUnit allowed developers to automate tests, create test suites, and integrate testing into their development workflow. Its introduction contributed significantly to raising awareness about the importance of testing and promoting the adoption of unit testing in the industry.\nThe 2000s: Agile Methods and Test-Driven Development (TDD) # In the early 2000s, agile development methodologies gained popularity, particularly Extreme Programming (XP) and Scrum. These methods emphasized short development cycles, continuous integration, and, most importantly, testing. One of the core ideas of XP is Test-Driven Development (TDD) , a methodology in which tests are created before actual code is written.\nIn TDD, the development process is controlled by tests:\n1. First, the developer writes a unit test that fails because the function to be tested still needs to be implemented.\n2. The minimal code required to pass the test is then written.\n3. Finally, the code is refined and optimised, with testing ensuring that existing functionality is not affected.\nTDD requires developers to engage intensively with unit testing, which drives the entire development process. Unit tests are not just a way to ensure quality but an integral part of the design.\nJUnit played a central role during this time, making implementing TDD in Java much more accessible. Developers could write and run tests quickly and easily, making the workflow more efficient.\nThe 2010s: Integration into CI/CD pipelines and the importance of automation # With the advent of Continuous Integration (CI) and Continuous Delivery (CD) In the 2010s, unit testing became even more critical. CI/CD pipelines allow developers to continuously integrate changes to the code base into the central repository and automatically deploy new versions of their software.\nIn a CI/CD environment, automated testing, including unit testing, is critical to ensure that new code changes do not break existing functionality. Unit tests are typically the first stage of testing executed in a CI pipeline because they are fast and focused.\nFor this purpose, numerous tools have been created that integrate unit tests into the CI/CD workflow. Besides JUnit, various build tools, like Maven and Gradle , can run unit tests automatically. Test reports are generated, and the developer is notified if a test fails.\nAnother significant advancement during this time was integrating code coverage tools like JaCoCo. These tools measure the percentage of code covered by unit tests and help developers ensure that their tests cover all relevant code paths.\nTools and frameworks for unit testing in Java # JUnit : Probably the best-known and most widely used testing framework for Java. Since its introduction in 1999, JUnit has evolved and offers numerous features that make testing easier. With the introduction of JUnit 5, the framework became more modular and flexible.\nTestNG : Another popular testing framework that offers similar functionality to JUnit but supports additional features such as dependency management and parallel test execution. TestNG is often used in larger projects that require complex testing scenarios.\nMockito : A mocking framework used to simulate dependencies of classes in unit tests. Mockito allows developers to “mock” objects to control the behaviour of dependencies without actually instantiating them.\nJaCoCo : A tool for measuring code coverage. It shows what percentage of the code is covered by tests and helps developers identify untested areas in the code.\nGradle and Maven : These build tools provide native support for unit testing. They allow developers to run tests and generate reports during the build process automatically.\nBest practices for unit testing in Java # Isolated testing : Each unit test should test a single method or function in isolation. This means that external dependencies such as databases, networks or file systems should be “mocked” to ensure that the test focuses only on the code under test.\nWrite simple tests : Unit tests should be simple and easy to understand. You should only test a single functionality and not have any complex logic or dependencies.\nRegular testing : Tests should be conducted regularly, especially before every commit or build. This ensures that errors are detected early and that the code remains stable.\nTest cases for limits : It is essential to test typical use cases and cover limit values ​​(e.g., minimum and maximum input values) and error cases.\nEnsure test coverage : Code coverage tools like JaCoCo help developers ensure that most of the code is covered by tests. However, high test coverage should be one of many goals. The quality of the tests is just as important as the quantity.\nThe future of unit testing # Unit testing has constantly evolved over the past few decades, and this development is expected to continue in the future. The advent of technologies like Artificial Intelligence (AI) and Machine Learning could open up new ways of generating and executing unit tests.\nAI-powered tools could automatically generate unit tests by analysing the code and suggesting tests based on typical patterns. This could further automate the testing process and give developers more time for actual development.\nAnother trend is the increasing integration of Mutation Testing. This technique involves making minor changes to the code to check whether the existing tests detect the mutated code paths and throw errors. This allows developers to ensure that their tests cover code and detect errors.\nUnit testing is an essential part of modern software development and has evolved from an optional practice to a standard. In Java, JUnit has contributed significantly to the proliferation and standardisation of unit testing. At the same time, agile methodologies such as TDD have integrated the role of testing into the development process.\nWith increasing automation through CI/CD pipelines and the availability of powerful tools, unit testing will continue to play a critical role in ensuring code quality in the future. Developers integrating unit testing into their workflow early on benefit from stable, maintainable and error-free code.\nWhat are the disadvantages of unit testing? # Unit testing has many advantages, especially ensuring code quality and finding errors early. However, there are also some disadvantages and limitations that must be taken into account in practice:\nTime expenditure and costs # Creation and maintenance : Writing unit tests requires additional time, especially in the early stages of a project. This effort increases development costs and can seem disproportionate for small projects.\nMaintenance : When the code changes, the associated tests often also need to be adapted. This can be very time-consuming for larger code bases. Changes to the design or architecture can lead to numerous test changes.\nLimited test coverage # Only tests isolated units : Unit tests test individual functions or methods in isolation. They do not cover the interaction of different modules or layers of the system, which means that they cannot reveal integration problems.\nNot suitable for all types of errors : Unit tests are good at finding logical errors in individual methods, but they cannot detect other types of errors, such as errors in interaction with databases, networks, or the user interface.\nFalse sense of security # High test coverage ≠ freedom from errors : Developers may develop a false sense of security when achieving high test coverage. Just because many tests cover the code doesn\u0026rsquo;t mean it\u0026rsquo;s bug-free. Unit tests only cover the specific code they were written and may not test all edge or exception cases.\nBlind trust in testing : Sometimes, developers rely too heavily on unit testing and neglect other types of testing, such as integration testing, system testing, or manual testing.\nExcessive mocking # Mocks can distort reality : When testing classes or methods that depend on external dependencies (e.g. databases, APIs, file systems), mock objects are often used to simulate these dependencies. However, extensive mocking frameworks such as Mockito can lead to unrealistic tests that work differently than the system in a real environment.\nComplex dependencies : When a class has many dependencies, creating mocks can become very complicated, making the tests difficult to understand and maintain.\nDifficulties in testability of legacy code # Legacy-Code ohne Tests : Writing unit tests can be challenging and time-consuming in existing projects with older code (legacy code). Such systems may have been designed without testability in mind, making unit tests difficult to write.\nRefactoring necessary : Legacy code often needs to be refactored to enable unit testing, which can introduce additional risks and costs.\nNot suitable for complex test cases # Not suitable for end-to-end testing : Unit tests are designed to test individual units and are not intended to cover end-to-end test cases or user interactions. Such testing requires other types of testing, such as integration, system, or acceptance testing.\nLimited perspective : Unit tests often only consider individual system components and not the behaviour of the entire system in real usage scenarios.\nTest-driven development (TDD) can lead to excessive focus on details # Design of code influenced by testing : In TDD, the emphasis is on writing tests before writing the code. This can sometimes lead to developers designing code to pass tests rather than developing a more general, robust solution.\nExcessive focus on detailed testing : TDD can cause developers to focus too much on small details and isolated components instead of considering the overall system architecture and user needs.\nTest maintenance for rapid changes # Frequent changes lead to outdated tests : In fast-paced development projects where requirements and code change frequently, tests can quickly become obsolete and unnecessary. Maintaining these tests can become significant without providing any clear added value.\nTests as ballast : If code is constantly evolving and tests are not updated, outdated or irrelevant tests can burden the development process.\nLack of testing strategies in complex systems # Complexity of test structure : It can be difficult to develop a meaningful unit testing strategy that covers all aspects of very complex systems. This often results in fragmented and incomplete tests or inadequate testing of critical areas of the system.\nTesting complexity in object-oriented designs : For highly object-oriented programs, it can be difficult to identify precise units for unit testing, especially if the classes are highly interconnected. In such cases, writing unit tests can become cumbersome and inefficient.\nAdditional effort without immediate benefit # Cost-benefit analysis for small projects : In small projects or prototypes where the code is short-lived, the effort spent writing unit tests may outweigh the benefits. In such cases, other testing methods, such as manual or simple end-to-end testing, may be more efficient.\nConclusion # Although unit testing has numerous advantages, there are also clear disadvantages that must be considered. The additional time required, limited test coverage, and potential maintenance challenges must be weighed when planning a testing process. Unit testing is not a panacea but should be viewed as one tool among many in software development. However, combined with other types of testing and a well-thought-out testing strategy, unit testing can significantly improve code quality.\nDoes Secure Pay Load Testing belong to the area of ​​unit testing? # Secure Payload Testing usually belongs to a different area than the traditional area of Unit Testing, which is Security testing and partial Integration Tests. Let\u0026rsquo;s take a closer look at this to better understand the boundaries.\nWhat is Secure Payload Testing? # Secure Payload Testing refers to testing security-related data or messages exchanged between system components. This particularly applies to scenarios where sensitive data (such as passwords, API keys, encrypted data, etc.) must be protected and handled correctly in communication between systems. It tests whether data is appropriately encrypted, decrypted and authenticated during transmission and whether there are any potential security gaps in handling this data.\nExamples of typical questions in secure payload testing are:\nIs sensitive data encrypted and decrypted correctly? Does data remain secure during transmission? Is it ensured that the payload data does not contain security holes, such as SQL injections or cross-site scripting (XSS)? Is the integrity of the payload guaranteed to prevent manipulation? Difference between Unit Testing and Secure Payload Testing # Unit Testing focuses on the Functionality of individual program components or methods in isolation**.** It usually checks whether a method delivers the expected outputs for certain inputs. The focus is on the correctness and stability of the program\u0026rsquo;s logical units, not directly on the security or protection of data.\nAn example of a unit test in Java would be testing a simple mathematical function. The unit test would check whether the method works correctly. Security aspects, such as handling confidential data or ensuring encryption, are usually outside of such tests.\nIn contrast, Secure Payload Testing involves the secure handling and processing of data during transmission or storage. This is often part of security testing , which aims to ensure that data is properly protected and not vulnerable to attacks or data leaks.\nWhere does Secure Payload Testing fit in? # Integration tests : Secure Payload Testing could be part of integration testing, which tests the interaction between different components of a system, e.g., between a client and a server. Here, one would ensure that the payload is properly encrypted and the transmission over the network is secure.\nSecurity testing : In more complex systems, secure payload testing belongs more to Security testing , which involves attacks on data security, integrity, and confidentiality of payloads. These tests often go beyond the functionality of individual code units and require special testing strategies, such as penetration testing or testing for known security vulnerabilities.\nEnd-to-End Tests : Since Secure Payload Testing is often related to data transfer, it can also be part of End-to-End tests. The entire system is tested here, from input to processing to output. These tests check whether the data is encrypted correctly at the beginning and decrypted and processed correctly at the end.\nCan Secure Payload Testing be part of Unit Testing? # In special cases, an aspect of secure payload testing can be part of unit tests, especially if the security logic is very closely linked to the functionality of the unit under test (e.g., an encryption or decryption method).\nAn example could be testing an Encryption method in isolation:\npublic String encrypt(String data, String key) { // Encryption logic return encryptedData; } public String decrypt(String encryptedData, String key) { // Decryption logic return decryptedData; } Here, you could write unit tests that check whether:\nThe text is correctly encrypted and decrypted. For the same input data, the same output is always produced (in the case of deterministic encryption). The method responds correctly to invalid inputs (e.g. wrong key, invalid data format). Despite these specific test cases, secure payload testing generally focuses on not only isolated functionality but also security and integrity in the context of other system components.\nSecure Payload Testing does not belong to classic Unit Testing. It typically concerns security, integrity, and correct data processing in a broader context, often involving the interaction of multiple system components. It falls more into the realm of security and integration tests.\nHowever, unit tests can test parts of security logic to some extent, especially when encryption or security functions are to be tested in isolation. However, the overall picture of security requirements and protection of payloads is usually determined by more comprehensive testing, such as Integration Tests , Security testing, or End-to-End tests.\nWhich secure coding practices play a role in connection with unit testing? # Secure Coding Practices are essential to ensure the code is secure against potential attacks and vulnerabilities. These practices are fundamental to ensuring software security, and they can be closely related toUnit Testing. While unit testing primarily aims to verify code\u0026rsquo;s functionality, secure coding practices can help ensure that code is robust and secure. Here are some of the key Secure Coding Practices related to Unit Testing:\nInput validation and sanitisation # Safe practice : Always ensure that input from external sources (user input, API calls, file input) is validated and “sanitised” to avoid dangerous content such as SQL injection or cross-site scripting (XSS).\nConnection to unit testing : Unit tests should ensure that methods and functions respond correctly to invalid or potentially dangerous input. Tests should contain inputs such as unexpected special characters, entries that are too long or short, empty fields, or formatting errors.\nExample unit test in Java :\n@Test public void testValidateUserInput() { String invalidInput = \u0026#34;\u0026lt;script\u0026gt;alert(\u0026#39;xss\u0026#39;)\u0026lt;/script\u0026gt;\u0026#34;; assertFalse(InputValidator.isValid(invalidInput)); } This test checks whether the isValid method correctly rejects unsafe input as invalid.\nLimit value analysis (boundary testing) # Safe practice : Inputs should be tested against their maximum and minimum limits to ensure the code does not crash or be vulnerable to buffer overflows.\nConnection to unit testing : Unit tests should ensure the application safely responds to input at the top and bottom of its allowed ranges. This helps prevent typical attacks such as buffer overflows.\nExample : If a function only accepts a certain number of characters, the unit test should check how the function responds to inputs that are precisely at or above that limit.\nSecure error handling # Safe practice : Error handling should not reveal sensitive information, such as stack traces or details about the application\u0026rsquo;s internal structure, as attackers can exploit such information.\nConnection to unit testing : Unit tests should ensure errors and exceptions are handled correctly without exposing sensitive information. Unit tests can trigger targeted exception situations and check whether only safe and user-friendly error messages are returned.\nExample :\n@Test public void testHandleInvalidInputGracefully() { try { myService.processUserInput(null); } catch (Exception e) { fail(\u0026#34;Exception thrown, but should have been handled gracefully.\u0026#34;); } } Avoiding hard-coded secrets # Safe practice : Never hard-code sensitive information such as passwords, API keys, or tokens in code. Instead, such data should be stored in secure environment variables or configuration files.\nConnection to unit testing : Unit tests should ensure that sensitive data is handled securely. They should also check that the code loads external configuration sources correctly and does not accidentally use hard-coded secrets.\nExample :\n@Test public void testExternalConfigForSecrets() { assertNotNull(System.getenv(\u0026#34;API_KEY\u0026#34;)); } Use of safe libraries and dependencies # Safe practice : Users should take care to use secure libraries and frameworks and update them regularly to avoid known security vulnerabilities.\nConnection to unit testing : Unit tests should ensure the libraries are correctly integrated and updated. Testing functionality that depends on external libraries is also essential to ensure that security mechanisms in those libraries are used correctly.\nEnsure encryption # Safe practice : Sensitive data should be stored and transmitted in encrypted form to prevent unauthorised access or data leaks.\nConnection to unit testing : Unit tests should verify that data is encrypted and decrypted correctly. For example, a test could ensure that the encryption and decryption methods work consistently and without errors.\nExample :\n@Test public void testEncryptionAndDecryption() { String original = \u0026#34;Sensitive Data\u0026#34;; String encrypted = encrypt(original, \u0026#34;mySecretKey\u0026#34;); String decrypted = decrypt(encrypted, \u0026#34;mySecretKey\u0026#34;); assertEquals(original, decrypted); } Least Privilege Principle # Safe practice : Methods and functions should only be executed with the minimum rights and access required.\nConnection to unit testing : Unit tests should ensure that methods only work with the minimum required data and that no unauthorised access to resources is possible. For example, tests could check whether protected resources are only accessed after successful authentication.\nExample :\n@Test public void testUnauthorizedAccess() { assertThrows(AccessDeniedException.class, () -\u0026gt; { userService.deleteUserDataWithoutPermission(); }); } Avoiding race conditions # Safe practice : Race conditions can occur when multiple threads or processes access shared resources simultaneously. They should be avoided to prevent security issues such as unpredictable behavior or data corruption.\nConnection to unit testing : Unit tests should ensure that the code is thread-safe and that no race conditions occur. This can be verified by testing code under multiple access or by using mock threads.\nExample :\n@Test public void testConcurrentAccess() throws InterruptedException { CountDownLatch latch = new CountDownLatch(2); Runnable task = () -\u0026gt; { sharedResource.modify(); latch.countDown(); }; Thread t1 = new Thread(task); Thread t2 = new Thread(task); t1.start(); t2.start(); latch.await(); assertTrue(sharedResource.isConsistent()); } Avoidance of buffer overflows # Safe practice : Buffer overflows occur when a program writes more data into a memory area than it can hold. Although Java is less prone to buffer overflows than C or C++, thanks to automatic memory management, care should still be taken to ensure that arrays and memory structures are used safely.\nConnection to unit testing : Unit tests should test edge cases and maximum input values ​​to ensure that overflows do not occur.\nSafe use of third-party libraries # Safe practice : Third-party libraries should be used safely and regularly checked for known vulnerabilities.\nConnection to unit testing : Tests can ensure that functions and classes from third-party libraries are implemented correctly and used safely. Mocking can be used to simulate external dependencies safely.\nSecure Coding Practices play an essential role in Unit Testing , as they ensure that the code is not only functionally correct but also secure against potential attacks. Unit tests should aim to check security-relevant aspects such as input validation, secure error handling, encryption, and rights assignment. By incorporating these practices into the unit testing process, developers can ensure their applications are robust, secure, and protected against many common attack patterns.\n","date":"16 October 2024","externalUrl":null,"permalink":"/posts/code-security-through-unit-testing-the-role-of-secure-coding-practices-in-the-development-cycle/","section":"Posts","summary":"Unit testing is an essential software development concept that improves code quality by ensuring that individual units or components of a software function correctly. Unit testing is crucial in Java, one of the most commonly used programming languages. This article will discuss what unit testing is, how it has evolved, and what tools and best practices have been established over the years.\n","title":"Code security through unit testing: The role of secure coding practices in the development cycle","type":"posts"},{"content":"","date":"7 October 2024","externalUrl":null,"permalink":"/tags/cwe-377/","section":"Tags","summary":"","title":"CWE-377","type":"tags"},{"content":"","date":"7 October 2024","externalUrl":null,"permalink":"/tags/java-nio/","section":"Tags","summary":"","title":"Java NIO","type":"tags"},{"content":"","date":"7 October 2024","externalUrl":null,"permalink":"/tags/toctou/","section":"Tags","summary":"","title":"TOCTOU","type":"tags"},{"content":"Building on the discussion of “CWE-377: Insecure Temporary File”, it’s essential to delve deeper into one of the most insidious vulnerabilities that can arise in this context—TOCTOU (Time-of-Check to Time-of-Use) race conditions. TOCTOU vulnerabilities occur when there is a time gap between verifying a resource (such as a file) and its subsequent use. Malicious actors can exploit this gap, especially in temporary file scenarios, leading to serious security breaches. This follow-up article will explore how TOCTOU conditions manifest in software, particularly in managing temporary files, and discuss strategies to mitigate these risks to ensure robust and secure application development.\nI wrote an article about CWE-377 itself. You can find it here: https://svenruppert.com/2024/08/21/cwe-377-insecure-temporary-file-in-java/\nTOCTOU (Time-of-Check to Time-of-Use) is a type of race condition that occurs when the state of a resource (such as a file, memory, or variable) is checked (validated or verified) and then used (modified or accessed) in separate steps. If an attacker can alter the resource between these two steps, they may exploit the gap to introduce malicious behaviour or compromise the security of an application.\nHow TOCTOU Applies to Temporary Files # In the context of temporary file creation, TOCTOU vulnerabilities arise when the program checks whether a temporary file exists and then creates or opens it. If an attacker manages to create a file with the same name in the interval between these operations, they can control the contents or properties of the file the program thinks it is safely creating or accessing.\nFor example, consider the following sequence of operations:\nCheck if a file with a specific name exists : The program checks if a temporary file (e.g., tempfile.txt) already exists.\nCreate or open the file : If the file does not exist, the program creates or opens it.\nIf an attacker creates a file named tempfile.txt in the time between the check and the creation, the program may inadvertently interact with the attacker\u0026rsquo;s file instead of a legitimate, secure file. This can lead to issues such as unauthorised data access, corruption, or privilege escalation.\nDetailed Example of TOCTOU Vulnerability # Consider the following Java code:\nimport java.io.File; import java.io.IOException; public class TOCTOUVulnerabilityExample { public static void main(String[] args) throws IOException { File tempFile = new File(\u0026#34;/tmp/tempfile.txt\u0026#34;); // Time-of-check: Verify if the file exists if (!tempFile.exists()) { // Time-of-use: Create the file tempFile.createNewFile(); System.out.println(\u0026#34;Temporary file created at: \u0026#34; + tempFile.getAbsolutePath()); } } } In this example:\n1. The program first checks if **tempfile.txt** exists using the **exists()** method.\n2. If the file does not exist, it creates a new file with the same name using the **createNewFile()** method.\nThe vulnerability here lies between the time the **exists()** check is performed and the time the **createNewFile()** method is called. If an attacker creates a file named **tempfile.txt** between these two steps, the program will not create a new file but instead interact with the attacker\u0026rsquo;s file, potentially leading to a security breach.\nExploitation of TOCTOU in Temporary Files # An attacker can exploit TOCTOU in several ways:\nFile Pre-Creation : The attacker creates a file with the same name as the intended temporary file in a directory accessible by the application. If the file permissions are weak, the attacker may gain control over the contents of this file.\nSymlink Attack : The attacker can create a symbolic link (symlink) that points to a sensitive file (e.g., /etc/passwd) with the same name as the temporary file. When the program tries to write to or read from the temporary file, it might access the sensitive file instead, leading to data corruption or information leakage.\nPrivilege Escalation : If the program runs with elevated privileges (e.g., as root), an attacker could exploit the TOCTOU race condition to modify files or data that they otherwise would not have permission to access or change.\nPreventing TOCTOU Vulnerabilities in Java # To prevent TOCTOU vulnerabilities, particularly when dealing with temporary files, developers should follow best practices that minimise the risk of a race condition:\nUse Atomic Operations\nAtomic operations are inseparable; they either complete entirely or do not happen at all, leaving no opportunity for an attacker to intervene. Java\u0026rsquo;s File.createTempFile() method is atomic when creating temporary files. This means that the file creation and name generation occur in a single step, eliminating the TOCTOU window.\nimport java.io.File; import java.io.IOException; public class AtomicTempFileCreation { public static void main(String[] args) throws IOException { // Atomic operation to create a temporary file File tempFile = File.createTempFile(\u0026#34;tempfile_\u0026#34;, \u0026#34;.tmp\u0026#34;); tempFile.deleteOnExit(); System.out.println(\u0026#34;Secure temporary file created at: \u0026#34; + tempFile.getAbsolutePath()); } } Here, **File.createTempFile()** ensures that the file is both uniquely named and securely created without exposing the application to race conditions.\nUse Secure Directories\nPlace temporary files in a secure, private directory that is inaccessible to other users. This limits attackers\u0026rsquo; opportunities to exploit TOCTOU vulnerabilities because they cannot easily place or manipulate files in these directories.\nLeverage Files and Path (NIO.2 API)\nJava’s NIO.2 API (java.nio.file) offers more advanced file-handling mechanisms, including atomic file operations. For instance, **Files.createTempFile()** allows for atomic file creation with customisable file attributes, such as secure permissions, further reducing the risk of TOCTOU vulnerabilities.\nimport java.nio.file.Files; import java.nio.file.Path; import java.nio.file.attribute.PosixFilePermissions; import java.util.Set; public class SecureAtomicTempFile { public static void main(String[] args) throws IOException { // Create a temporary file with atomic operations and secure permissions Path tempFile = Files.createTempFile(\u0026#34;secure_tempfile_\u0026#34;, \u0026#34;.tmp\u0026#34;, PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString(\u0026#34;rw-------\u0026#34;))); System.out.println(\u0026#34;Secure temporary file created at: \u0026#34; + tempFile.toAbsolutePath()); } } This approach combines atomic file creation with restrictive file permissions, mitigating both TOCTOU vulnerabilities and other potential security risks.\nConclusion # TOCTOU vulnerabilities represent a significant security risk when handling temporary files, particularly if these files are created in an insecure or non-atomic manner. The key to preventing these vulnerabilities lies in eliminating the gap between the time-of-check and time-of-use, typically by using atomic file creation methods provided by secure APIs, such as **File.createTempFile()** or **Files.createTempFile()** .\nBy understanding the risks associated with TOCTOU race conditions and following best practices, developers can ensure that their Java applications are resilient to these attacks and maintain the software\u0026rsquo;s integrity and security.\n","date":"7 October 2024","externalUrl":null,"permalink":"/posts/understanding-toctou-time-of-check-to-time-of-use-in-the-context-of-cwe-377/","section":"Posts","summary":"Building on the discussion of “CWE-377: Insecure Temporary File”, it’s essential to delve deeper into one of the most insidious vulnerabilities that can arise in this context—TOCTOU (Time-of-Check to Time-of-Use) race conditions. TOCTOU vulnerabilities occur when there is a time gap between verifying a resource (such as a file) and its subsequent use. Malicious actors can exploit this gap, especially in temporary file scenarios, leading to serious security breaches. This follow-up article will explore how TOCTOU conditions manifest in software, particularly in managing temporary files, and discuss strategies to mitigate these risks to ensure robust and secure application development.\n","title":"Understanding TOCTOU (Time-of-Check to Time-of-Use) in the Context of CWE-377","type":"posts"},{"content":"","date":"26 September 2024","externalUrl":null,"permalink":"/tags/bld/","section":"Tags","summary":"","title":"Bld","type":"tags"},{"content":" What is a dependency management tool? # A dependency management tool is a software system or utility that automates the process of identifying, retrieving, updating, and maintaining the external libraries or packages (referred to as dependencies) required by a software project. It ensures that all necessary dependencies are included and managed in a standardised way, which helps prevent version conflicts, missing libraries, and manual errors during software development.\nWhat is a dependency management tool? Dependency Identification: Fetching Dependencies: Version Management: Scope \u0026amp; Environment Management: Build Process Integration: Popular Dependency Management Tools: Benefits of Dependency Management Tools: Example Workflow: The classical Dependency manager for Java - Maven Project Structure and pom.xml Build Lifecycle Dependency Management Maven Repositories Plugins How It All Works Together: BLD - the lightweight Java Build System Key Features of bld How bld Stands Out in the Java Ecosystem Usage Example The difference between Maven and BLD Build Language: Declarative vs. Code-Based: Task Execution: IDE Integration: Dependency Management: Build Artifacts: Complexity and Learning Curve: Performance: How to start with bld? An Example migration of a project from Maven to bld The Maven Project Structure Conclusion Demo - Project are available here: https://github.com/Java-Publications/Blog---Core-Java---2024.08---bld-the-lightweight-build-tool\nHere’s how dependency management tools generally function:\nDependency Identification: # Developers list the libraries or frameworks their project depends on in a configuration file (like Maven’s pom.xml or Gradle’s build.gradle). Each dependency is defined by attributes such as group ID , artifact ID , and version.\nExample (Maven) :\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;junit\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;junit\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;4.12\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; Fetching Dependencies: # The tool retrieves the required dependencies from repositories (such as Maven Central or custom/private repositories) and downloads them to a local repository on the developer’s machine. These repositories host many external libraries, frameworks, and plugins.\nLocal Repository : A cache stored locally on a developer’s machine to prevent repeated downloads.\nRemote Repository : Centralized or custom servers hosted by the dependencies (e.g., Maven Central).\nVersion Management: # Dependency management tools handle dependency versioning , ensuring the correct versions of external libraries are used. Tools like Maven or Gradle support transitive dependency resolution , meaning if a dependency relies on another library, that secondary library will also be fetched automatically.\nVersion Conflicts : If two libraries depend on different versions of the same dependency, the tool resolves the conflict by selecting the nearest version in the dependency tree or the explicitly defined one.\nScope \u0026amp; Environment Management: # Dependency management tools allow for different scopes or profiles that specify when a dependency is required (e.g., only during testing or in a production environment). This helps optimise resource usage and prevent unnecessary dependencies from being included in the final build.\nExample Scopes :\nCompile : Dependencies available during compilation and runtime.\nTest : Dependencies used only during testing (e.g., JUnit).\nProvided : Dependencies expected to be supplied by the runtime environment.\nBuild Process Integration: # Dependency management tools integrate into a project\u0026rsquo;s build lifecycle. They ensure that the right versions of libraries are fetched before compiling, packaging, or testing the application. For instance, Maven\u0026rsquo;s **install** phase ensures that all dependencies are installed in the local repository before building.\nPopular Dependency Management Tools: # Maven : Uses **pom.xml** to define dependencies and follows a declarative model with predefined lifecycles.\nGradle : Uses Groovy or Kotlin DSL for its build scripts, offering more flexibility and allowing for custom build logic.\nnpm (Node Package Manager) : Manages dependencies for JavaScript and Node.js projects, using a **package.json** file.\nbld : A newer tool for Java projects that allows developers to define dependencies directly in Java code.\nBenefits of Dependency Management Tools: # Automation : They automate the retrieval and management of libraries, removing the need for developers to download and include them manually.\nConsistency : Ensures that every developer in a team uses identical versions of dependencies, leading to more consistent builds.\nConflict Resolution : Handles version conflicts between dependencies automatically, preventing runtime errors.\nEfficiency : By caching dependencies locally, tools reduce repeated downloads and improve build times.\nSecurity : Modern tools often check for security vulnerabilities in dependencies and alert developers to updates.\nExample Workflow: # A developer declares dependencies in a configuration file.\nThe tool checks the local repository for cached versions of the libraries.\nIf dependencies aren’t cached, the tool fetches them from a remote repository.\nThe tool ensures all dependencies are included during the build process (compile, test, package, etc.).\nDependency management tools are vital for modern software development, especially in large projects with numerous external libraries. They automate the complex process of handling dependencies, reducing manual errors, improving efficiency, and ensuring the project builds correctly across different environments.\nThe classical Dependency manager for Java - Maven # Maven is a build automation and project management tool primarily used in Java projects. It uses a declarative model to manage a project’s build, reporting, and dependencies using an XML configuration file called **pom.xml** . Below is an overview of how Maven works:\nProject Structure and pom.xml # At the heart of every Maven project is the **pom.xml** file (Project Object Model). This file defines the project structure, dependencies, build plugins, and goals. Here\u0026rsquo;s what **pom.xml** generally contains:\nGroup ID : Identifies the project’s group (often based on package naming).\nArtifact ID : A unique identifier for the project.\nVersion : Version of the project being built.\nDependencies : A list of libraries or frameworks the project requires (such as **JUnit** for testing or **Spring** for dependency injection).\nPlugins : Additional tools that enhance the build process (for tasks like creating JAR files, running tests, etc.).\n\u0026lt;project\u0026gt; \u0026lt;modelVersion\u0026gt;4.0.0\u0026lt;/modelVersion\u0026gt; \u0026lt;groupId\u0026gt;com.example\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;my-app\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.0-SNAPSHOT\u0026lt;/version\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;junit\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;junit\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;4.12\u0026lt;/version\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; \u0026lt;/project\u0026gt; Build Lifecycle # Maven operates on a series of lifecycles. The default lifecycle consists of a sequence of build phases that are invoked when building a project:\nvalidate : Check if the project is correct and all necessary information is available. compile : Compile the source code of the project. test : Run tests on the compiled code using a suitable testing framework (e.g., JUnit). package : Package the compiled code into a distributable format, like a JAR or WAR. install : Install the package into the local repository for use as a dependency in other local projects. deploy : Copy the final package to the remote repository for sharing with other developers. These lifecycle phases are sequential, meaning invoking **install** will also execute all phases before it (from **validate** to **package** ).\nDependency Management # Maven’s most powerful feature is its dependency management system. Maven fetches external libraries from Maven Central , a global repository that hosts many open-source libraries. Maven uses transitive dependency resolution , meaning that if Project A depends on Library B and Library B depends on Library C, Maven will automatically fetch Library C.\nDependencies are defined in the **pom.xml** under the **\u0026lt; dependencies\u0026gt;** section, as shown above. Maven manages and can automatically update versions, making integrating libraries very efficient.\nMaven Repositories # Maven uses three types of repositories:\nLocal Repository : This is on the developer’s machine. When Maven builds a project, it checks the local repository first to resolve dependencies.\nCentral Repository : Maven Central is the default remote repository where Maven looks for dependencies that are not available in the local repository.\nRemote Repository : Additional remote repositories, such as private or custom repositories that host specific artefacts, can be defined.\nPlugins # Plugins in Maven extend its functionality. Some common Maven plugins include:\nmaven-compiler-plugin : Compiles the source code.\nmaven-surefire-plugin : Runs unit tests.\nmaven-jar-plugin : Packages the project as a JAR file.\nmaven-war-plugin : Packages the project as a WAR file for web applications.\nThese plugins are usually added to the **pom.xml** under the **\u0026lt; build\u0026gt;** section.\nProfiles\nMaven also supports profiles, which allow developers to define different configurations for different environments (e.g., development, testing, production). Profiles are specified in the pom.xm l and can be activated by different triggers, such as system properties or environments.\n\u0026lt;profiles\u0026gt; \u0026lt;profile\u0026gt; \u0026lt;id\u0026gt;dev\u0026lt;/id\u0026gt; \u0026lt;properties\u0026gt; \u0026lt;env\u0026gt;development\u0026lt;/env\u0026gt; \u0026lt;/properties\u0026gt; \u0026lt;/profile\u0026gt; \u0026lt;/profiles\u0026gt; Multi-Module Projects\nMaven can handle multi-module projects , where a single project is divided into multiple sub-projects or modules. Each module typically has its own **pom.xml** but is coordinated by a parent POM. This feature is helpful for large projects with independent yet related components.\nHow It All Works Together: # 1. The developer defines the project structure and dependencies in the pom.xml.\n2. Maven processes this file and resolves all dependencies by downloading required JAR files from remote repositories (if they are not found in the local repository).\n3. Maven executes phases from the build lifecycle, compiling, testing, and packaging the project.\n4. Plugins add extra functionality, such as running tests, generating documentation, and creating artefacts.\nMaven’s declarative model and lifecycle management make it a popular choice for managing complex Java projects. Its rich plugin ecosystem and dependency management reduce manual effort while standardising the build process. However, the XML-heavy configuration can be cumbersome for some developers, especially compared to newer build tools like bld.\nBLD - the lightweight Java Build System # bld is a lightweight build tool designed specifically for the Java ecosystem. Unlike traditional build tools like Maven or Gradle, which rely on declarative syntax and often involve complex configurations, bld allows developers to write their build logic in pure Java. This approach simplifies the learning curve and reduces the cognitive load on developers by enabling them to stay within the Java language for both application and build logic, making it a particularly modern and streamlined tool for Java projects.\nKey Features of bld # Java-Based Build Logic : The most distinctive aspect of bld is that it uses Java itself to define the build process. This eliminates the need for learning domain-specific languages (DSLs) or XML configurations as is the case with Maven or Gradle. With bld , build logic becomes part of the Java code, making it easy to reason about and maintain in the same environment where application development occurs.\nExplicit Task Execution : bld emphasises a transparent and predictable execution model. Tasks in bld do not run automatically but must be explicitly triggered, giving developers complete control over the build process. This removes any \u0026ldquo;auto-magical\u0026rdquo; behaviour often associated with other tools, which can sometimes lead to unexpected outcomes.\nSimple Dependency Management : Dependency management in bld is intuitive and leverages Java constructs. Dependencies are managed through a standard library system, and they can be added directly in the Java build script by specifying the Maven coordinates. This system allows bld to handle fetching and resolving dependencies automatically, similar to Maven or Gradle, but with less complexity.\nIntegration with Java 17 : bld takes full advantage of the latest Java features, relying on Java 17 as its minimum requirement. This allows developers to use modern Java syntax, including records, sealed classes, and pattern matching, to write concise and clean build scripts. The tool supports features like code auto-completion and Javadoc integration, making it easy to navigate within popular Integrated Development Environments (IDEs).\nIDE Support : bld has integrations for popular IDEs like IntelliJ IDEA, offering features such as automatic project detection, quick access to the main build class, and the ability to run multiple build commands directly from the IDE. This smooths the workflow for developers who prefer using graphical interfaces over command-line interactions.\nCommand-Line Interface (CLI) : The bld CLI offers a set of standard commands that allow developers to compile, run, test, and package their projects. It supports everyday build tasks like creating JAR files, managing dependencies, running tests, and generating Javadoc. Developers can also define custom commands within their build scripts using Java, extending the tool’s flexibility.\nHow bld Stands Out in the Java Ecosystem # In the Java build ecosystem, tools typically fall into two categories: declarative (like Maven) and code-based (like Gradle). bld is part of the second group but adds the advantage of being immediately executable without predefining build plans. This immediate execution reduces the complexity of managing and understanding the build pipeline.\nFor developers familiar with scripting environments like Node.js or Go, bld \u0026rsquo;s ergonomics offer a familiar, streamlined experience. It delivers the ease of dependency management while maintaining transparency and direct control, helping to avoid common issues associated with over-engineered build systems.\nUsage Example # A basic bld file for a Java project could look like this:\npackage com.example; import rife.bld.Project; import java.util.List; import static rife.bld.dependencies.Repository.*; import static rife.bld.dependencies.Scope.*; public class MyappBuild extends Project { public MyappBuild() { pkg = \u0026#34;com.example\u0026#34;; name = \u0026#34;Myapp\u0026#34;; mainClass = \u0026#34;com.example.MyappMain\u0026#34;; version = version(0, 1, 0); repositories = List.of(MAVEN_CENTRAL); scope(compile).include(dependency(\u0026#34;com.fasterxml.jackson.core\u0026#34;, \u0026#34;jackson-databind\u0026#34;, version(2, 16, 0))); } } In this setup, a developer specifies dependencies in the build file itself, alongside build configurations, removing the need for separate files like **pom.xml** in Maven.\nbld offers a powerful yet simple alternative for Java developers who seek to avoid the complexities of traditional build tools. Writing build logic in Java, using an explicit task execution model, and supporting modern Java features provide a clear and concise way to manage builds, dependencies, and distribution. It suits developers who prefer a code-centric approach to their entire development lifecycle, from coding to building.\nThe difference between Maven and BLD # bld and Maven are both Java build tools, but they differ significantly in their approach, complexity, and usage. Here’s a comparison of the two:\nBuild Language: # Maven : Uses an XML-based configuration file called **pom.xml** (Project Object Model). This declarative approach requires developers to define a static configuration of dependencies, plugins, and tasks in a verbose XML format.\nbld : It uses Java as the build scripting language, meaning developers write their build logic using Java code. This approach leverages Java\u0026rsquo;s advantages, such as type safety, auto-completion, and the ability to refactor using IDE tools.\nDeclarative vs. Code-Based: # Maven : Fully declarative. Developers declare what they want to happen, and Maven uses its lifecycle to determine how to execute tasks. This provides predictability but can be hard to customise beyond its standard behaviour.\nbld : Primarily code-based. Developers write Java code that is executed immediately. This provides more flexibility and control over the build process, allowing custom build flows and logic to be easily implemented.\nTask Execution: # Maven : Its lifecycle uses predefined phases, such as **validate** , **compile** , **test** , **package** , and **install** . Each phase automatically triggers a set of goals, which is convenient for standard projects but can become cumbersome if you need custom behaviour outside of Maven’s predefined lifecycle.\nbld : Offers explicit control over tasks. Tasks are not automatically linked in a lifecycle; developers must call them explicitly, which provides more control and avoids the hidden behaviour common in Maven.\nIDE Integration: # Maven : Has widespread support across IDEs like Eclipse, IntelliJ IDEA, and NetBeans. It has been around for a long time, so IDEs can automatically recognise Maven projects and provide support like dependency management, project configuration, and running lifecycle phases directly from the IDE.\nbld : Offers integration with IntelliJ IDEA and leverages the build logic as Java code, providing features such as code navigation, auto-completion, and even IDE-run configurations.\nDependency Management: # Maven : Dependency management is one of Maven’s most vital features. It uses a central repository (Maven Central) and follows a transitive dependency model, automatically downloading dependencies and their sub-dependencies. However, this sometimes results in dependency conflicts or versioning issues.\nbld : Similar to Maven, bld supports dependency management using Maven Central or other repositories. However, it allows developers to handle dependency logic within the Java build file itself, which can simplify certain aspects of dependency resolution.\nBuild Artifacts: # Maven : Offers a standardised way to build JAR, WAR, and other packages using plugins. It also supports additional artefact types like sources, documentation, and testing.\nbld : Supports similar build artifacts (JAR, UberJAR, etc.) but gives developers more direct control over how these are generated. For example, developers can include custom commands to create specific outputs like precompiled templates.\nComplexity and Learning Curve: # Maven : Can be complex, especially when dealing with multi-module projects or needing custom behaviour that requires a deep understanding of Maven’s plugin architecture and lifecycle phases. Modifying or extending Maven’s behaviour often involves writing custom plugins or extensive XML configuration.\nbld : Aims to reduce complexity by allowing developers to manage builds using familiar Java code. It is simpler for Java developers to grasp, and writing custom build logic feels more natural compared to learning Maven\u0026rsquo;s XML configuration and plugins.\nPerformance: # Maven : Due to its lifecycle model and plugin architecture, Maven can be slower, mainly when performing complete builds with many plugins or complex dependency graphs. Incremental builds and caching are not as optimised as some newer tools.\nbld : Provides faster builds in many cases due to its simpler execution model and the immediate execution of Java code without going through multiple lifecycle phases.\nHow to start with bld? # The easiest way to start with bld is on commandline. For this go into the directory where you want to start with the project. Open a terminal and execute the following command that is available at the home page of the project:\nbash -c \u0026#34;$(curl -fsSL https://rife2.com/bld/create.sh)\u0026#34; Downloading bld v2.1.0... Welcome to bld v2.1.0. Please enter a number for the project type: 1: base (Java baseline project) 2: app (Java application project) 3: lib (Java library project) 4: rife2 (RIFE2 web application) This will download the bld-Script and execute it. Now, you can choose what kind of project you want to start. I am selecting 1 for a baseline project. The next questions are package name and application name. With this informationes the generating process will be done. You will find a directory with the appname inside the directory. In my example it is the name core.\nInside the app directory there is a script for unix/osx and one for windows. This is the wrapper for bld. I am using the osx version in my example here.\nTo test if the installation is correct, use the command ./bld version. In my case the answer is 2.6.0.\nThere are two directories. The first one is called lib. Inside there are different directories, one is called bld and the others are named by the different scopes you know from the maven lifecycles. (compile , provided , runtime and test). Later you will find the dependencies that are required for the different scopes inside these directories. The most important directory right now, is the directory called bld. Inside the directory there is the build related files like the wrapper, properties and cache. For customizing the build environment, the properties file is the right place to go. Here you can activate extensions for example. But this will be dome later.\nBack at the root directory of the project, there is the src folder as well. Inside the source folder are three subfolders. The src and test folder is known from maven already. The third folder called bld contains the sources for the build configuration. The Class is called appname plus “Build” in this case it is CoreBuild.\nHere we can start defining the build process itself.\nAn Example migration of a project from Maven to bld # Migrating a Maven project to bld involves several steps, as bld operates on Java-based build scripts instead of Maven’s **pom.xml** XML-based configuration. Here\u0026rsquo;s a general example of how you might approach migrating a Maven project to bld.\nThe Maven Project Structure # Before migrating, you need to analyse the Maven project structure:\n**pom.xml** : This file contains dependencies, plugins, repositories, build profiles, and other configurations.\nWe will go through different sections of the pom.xml and transforming them into the bld version.\n\u0026lt;groupId\u0026gt;com.svenruppert\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;core\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.0.1-SNAPSHOT\u0026lt;/version\u0026gt; Start by creating a new bld build script in Java. bld uses pure Java code to define the build logic, dependencies, and configurations, allowing you to migrate the settings from the **pom.xml** . First, create a **Build.java** file (or similar) in your **src/bld/java/** directory, if not already existing. Here is a starting point for the migration:\npackage com.svenruppert; import rife.bld.Project; import rife.bld.dependencies.VersionNumber; import java.nio.file.Path; import java.util.List; public class CoreBuild extends Project { public CoreBuild() { pkg = \u0026#34;com.svenruppert\u0026#34;; name = \u0026#34;Core\u0026#34;; mainClass = \u0026#34;com.svenruppert.Application\u0026#34;; version = version(0, 1, 0,\u0026#34;SNAPSHOT\u0026#34;); public static void main(String[] args) { new CoreBuild().start(args); } } As next we are migrating the definition of the repositories.\n\u0026lt;repositories\u0026gt; \u0026lt;repository\u0026gt; \u0026lt;snapshots\u0026gt; \u0026lt;enabled\u0026gt;false\u0026lt;/enabled\u0026gt; \u0026lt;/snapshots\u0026gt; \u0026lt;id\u0026gt;central\u0026lt;/id\u0026gt; \u0026lt;name\u0026gt;libs-release\u0026lt;/name\u0026gt; \u0026lt;url\u0026gt;https://repo.maven.apache.org/maven2/\u0026lt;/url\u0026gt; \u0026lt;/repository\u0026gt; \u0026lt;repository\u0026gt; \u0026lt;snapshots\u0026gt; \u0026lt;enabled\u0026gt;true\u0026lt;/enabled\u0026gt; \u0026lt;updatePolicy\u0026gt;always\u0026lt;/updatePolicy\u0026gt; \u0026lt;/snapshots\u0026gt; \u0026lt;id\u0026gt;snapshots\u0026lt;/id\u0026gt; \u0026lt;name\u0026gt;libs-snapshot\u0026lt;/name\u0026gt; \u0026lt;url\u0026gt;https://repo.maven.apache.org/maven2/\u0026lt;/url\u0026gt; \u0026lt;/repository\u0026gt; \u0026lt;/repositories\u0026gt; This will lead to a statement like the following inside the Build-Class.\nrepositories = List._of_(_MAVEN_CENTRAL_ , _RIFE2_RELEASES_); For the most projects you will need some dependencies in different scopes. The xml - Version wil look like the following. I am not listing all the dependencies of the demo project.\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.javalin\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;javalin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;6.3.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.javalin\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;javalin-testtools\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;6.3.0\u0026lt;/version\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; The bld Version:\nVersionNumber versionJavalin = version(6, 3, 0); scope(compile) .include(dependency(\u0026#34;io.javalin\u0026#34;, \u0026#34;javalin\u0026#34;, versionJavalin)); scope(test) .include(dependency(\u0026#34;io.javalin\u0026#34;, \u0026#34;javalin-testtools\u0026#34;, versionJavalin)); Finally we a have to migrate the plugins from maven to bld. If your Maven project used custom lifecycle phases, you can create custom commands in bld using Java methods annotated with **@BuildCommand** . This allows you to replicate any special build steps that were part of your Maven process, like packaging, publishing, or precompilation tasks.\nExample of adding a custom command in bld :\n@BuildCommand public void customTask() { System.out.println(\u0026#34;Running custom task...\u0026#34;); } Here we are at a point, where we can see, that bld is not as feature ritch as maven out of the box. But, if you need support for another plugin or lifecycle, have a look at the documentation about extensions. Here you will find all you need to create one by yourself. The cool thing here is, that everything can be done on core java. Even complex tasks can be implemented including all technologies that are needed for this step.\nBut, back to the migration. In my case I am using the pitest plugin to generate the mutation test coverage reports.\nInside the pom.xml you have to declare the following:\n\u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.pitest\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;pitest-maven\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.14.1\u0026lt;/version\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.pitest\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;pitest-junit5-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.2.1\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;threads\u0026gt;2\u0026lt;/threads\u0026gt; \u0026lt;outputFormats\u0026gt; \u0026lt;outputFormat\u0026gt;XML\u0026lt;/outputFormat\u0026gt; \u0026lt;outputFormat\u0026gt;HTML\u0026lt;/outputFormat\u0026gt; \u0026lt;/outputFormats\u0026gt; \u0026lt;targetClasses\u0026gt; \u0026lt;param\u0026gt;${pitest-prod-classes}\u0026lt;/param\u0026gt; \u0026lt;!--\u0026lt;param\u0026gt;org.reflections.*\u0026lt;/param\u0026gt;--\u0026gt; \u0026lt;/targetClasses\u0026gt; \u0026lt;targetTests\u0026gt; \u0026lt;param\u0026gt;${pitest-test-classes}\u0026lt;/param\u0026gt; \u0026lt;/targetTests\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/plugin\u0026gt; The corresponding bld part will be:\nscope(test) .include(dependency(\u0026#34;org.pitest\u0026#34;, \u0026#34;pitest\u0026#34;, version(1, 17, 0))) .include(dependency(\u0026#34;org.pitest\u0026#34;, \u0026#34;pitest-command-line\u0026#34;, version(1, 17, 0))) .include(dependency(\u0026#34;org.pitest\u0026#34;, \u0026#34;pitest-junit5-plugin\u0026#34;, version(1, 2, 1))) .include(dependency(\u0026#34;org.pitest\u0026#34;, \u0026#34;pitest-testng-plugin\u0026#34;, version(1, 0, 0))); @BuildCommand(summary = \u0026#34;Run PIT mutation tests\u0026#34;) public void pit() throws Exception { new PitestOperation() .fromProject(this) .reportDir(Path.of(\u0026#34;reports\u0026#34;, \u0026#34;mutations\u0026#34;).toString()) .targetClasses(pkg + \u0026#34;.*\u0026#34;) .targetTests(\u0026#34;junit.\u0026#34; + pkg + \u0026#34;.*\u0026#34;) .verbose(true) .execute(); } Now, you are able to trigger the pitest extension on commandline with ./bld pitest. To activate the plugin there must be the declaration inside the file “lib/bld/bld-wrapper.properties ”. The value of this key is the plugin maven coordinates including the version number.\nbld.extensions=com.uwyn.rife2:bld-pitest:1.0.2 With this we are able to migrate maven project into bld-projects.\nConclusion # Migrating a complex Maven project to bld involves translating the dependency management, build logic, and testing configurations from pom.xml into Java code using bld ’s project class structure. While both tools manage dependencies and build processes, bld allows developers to write build logic in Java, giving more flexibility and making it easier to manage as part of the application codebase.\nWith bld , you gain direct control over the build process through Java code, but you must ensure that all elements of your Maven build (dependencies, plugins, phases) are correctly transferred.\nHappy Coding\nSven\n","date":"26 September 2024","externalUrl":null,"permalink":"/posts/bld-a-lightweight-java-build-tool/","section":"Posts","summary":"What is a dependency management tool? # A dependency management tool is a software system or utility that automates the process of identifying, retrieving, updating, and maintaining the external libraries or packages (referred to as dependencies) required by a software project. It ensures that all necessary dependencies are included and managed in a standardised way, which helps prevent version conflicts, missing libraries, and manual errors during software development.\n","title":"BLD - a lightweight Java Build Tool","type":"posts"},{"content":"","date":"26 September 2024","externalUrl":null,"permalink":"/tags/buildsystem/","section":"Tags","summary":"","title":"Buildsystem","type":"tags"},{"content":"","date":"25 September 2024","externalUrl":null,"permalink":"/tags/tapestry/","section":"Tags","summary":"","title":"Tapestry","type":"tags"},{"content":"","date":"25 September 2024","externalUrl":null,"permalink":"/tags/webcomponent/","section":"Tags","summary":"","title":"Webcomponent","type":"tags"},{"content":" Tapestry, Wicket, and Vaadin # A component-oriented Java web application framework is a development framework that enables the construction of web applications in a modular way, using reusable, encapsulated components that manage their state, behaviour, and presentation. This approach allows developers to build complex user interfaces by assembling pre-built or custom components, like building blocks, each handling specific functionalities within the application.\nTapestry, Wicket, and Vaadin What is the challenge in creating component-oriented web frameworks in Java? State Management Component Lifecycle Event Handling Performance Optimisation Seamless Integration with Client-Side Technologies Security Concerns Usability and Developer Productivity Scalability What is Apache Tapestry? Use Cases What is Apache Wicket? Key Features of Apache Wicket Use Cases Vaadin Plattform Vaadin Flow The Difference between Apache Tapestry, Apache Wicket and Vaadin Flow Apache Tapestry Apache Wicket Vaadin Flow Key Differentiators Encapsulation : Each component is a self-contained unit that includes its own logic and data handling. This encapsulation helps hide the internal details of components while exposing a defined interface with which other components can interact.\nReusability : Components are designed to be reused across different parts of an application or even across multiple projects. This can significantly speed up development and ensure consistency throughout the application.\nModularity : The application can be divided into smaller, manageable parts (components), which can be developed, tested, and maintained independently. This modularity makes it easier to manage complex applications and allows multiple developers to work on different components simultaneously without conflicts.\nInteractivity : Component-oriented frameworks often support event-driven programming, allowing components to react to user inputs or other system events. This makes it straightforward to build dynamic and interactive web applications.\nIntegration : These frameworks typically provide easy integration with other technologies, such as databases, web services, and third-party libraries, allowing components to interact with these resources smoothly.\nWhat is the challenge in creating component-oriented web frameworks in Java? # Creating component-oriented web frameworks in Java poses several challenges, primarily due to the complexities of web development and the nature of Java as a server-side technology. Here are some of the main challenges:\nState Management # Web applications are inherently stateless, meaning each request is independent of previous requests. Component-oriented frameworks need to manage the state across multiple requests and user sessions. This can be complex, particularly when considering the need to synchronise the state between the server and the client while ensuring performance and scalability.\nComponent Lifecycle # Managing the lifecycle of components—creation, initialisation, rendering, updates, and destruction—is a significant challenge. Each component must handle its lifecycle efficiently, interacting with other components and the broader application environment without memory leaks or performance bottlenecks.\nEvent Handling # In a typical desktop application, the event-driven model is straightforward because the application runs in a single environment. However, in web applications, the user interface runs in the browser (client-side), while the business logic often runs on the server. This separation requires a robust mechanism to handle events on the client side, process them on the server, and reflect the changes to the client, all without making the application feel sluggish to the user.\nPerformance Optimisation # Component-oriented frameworks can introduce overhead due to the need for fine-grained components to manage their state and behaviour. This can impact performance, as every user interaction might require server communication. Handling this communication and minimising the data transferred between client and server is crucial for maintaining a responsive user experience.\nSeamless Integration with Client-Side Technologies # While Java is used on the server side, the client side typically involves HTML, CSS, and JavaScript. Ensuring that Java components integrate seamlessly with these technologies and that the framework can support rich client-side interactions is challenging. This includes providing support for Ajax and JavaScript callbacks and updating UI parts without refreshing the entire page.\nSecurity Concerns # Component-oriented frameworks need to manage security at the component level, ensuring that components do not expose vulnerabilities (such as cross-site scripting (XSS) or cross-site request forgery (CSRF)) and that the application can securely manage user data and authentication across components.\nUsability and Developer Productivity # The framework should be robust, flexible, and easy to use. Providing a high level of abstraction while allowing developers enough control to customise component behaviour and appearance is a delicate balance. Documentation, tooling, and community support are crucial for developer adoption and productivity.\nScalability # As web applications grow, they must efficiently handle increasing loads and concurrent users. Component-oriented frameworks must ensure that components do not become bottlenecks and that the application can scale horizontally across multiple servers or instances.\nAddressing these challenges requires a thoughtful design of the framework\u0026rsquo;s architecture, focusing on efficiency, flexibility, and ease of use to make the development of complex web applications feasible and efficient.\nWhat is Apache Tapestry? # Apache Tapestry is an open-source component-based web application framework for Java. It is designed to simplify the development of complex web applications by providing a highly scalable, efficient, and expressive programming model. Tapestry focuses on high productivity and maintainable code, adhering to the \u0026ldquo;convention over configuration\u0026rdquo; principle, which minimises the need for extensive XML configuration files commonly seen in older Java frameworks.\nComponent-Based Architecture : Tapestry structures applications as a collection of components. Each component has its template (typically an HTML file) and an optional Java class that manages its behaviour. This model encourages reusable UI segments and separation of concerns.\nLive Class Reloading : One of Tapestry\u0026rsquo;s standout features is its ability to reload Java classes automatically whenever they are changed. This means developers can modify the code and see the effects immediately in the browser without restarting the server, enhancing developer productivity.\nConvention Over Configuration : Tapestry reduces the need for explicit configuration by following convention over configuration. For example, it automatically detects templates based on naming conventions. This approach minimises boilerplate and setup code, making projects easier to manage and quicker to develop.\nHigh Performance : Tapestry includes built-in support for aggressive caching and minimisation of server-side processing, which enhances the application\u0026rsquo;s performance. It generates compact, efficient JavaScript code and CSS, optimising applications\u0026rsquo; load time and runtime performance.\nInbuilt Ajax Support : Tapestry supports Ajax, allowing developers to quickly update parts of the page and manage client-server communication asynchronously. This is handled in a way that abstracts much of the complexity associated with Ajax in raw JavaScript.\nPowerful Templating : Each Tapestry component can have its HTML template, allowing designers and developers to work together seamlessly. The framework automatically links HTML templates with their corresponding Java classes, providing a robust foundation for clean, maintainable code.\nDependency Injection and IoC Container : Tapestry integrates a powerful Inversion of Control (IoC) container that manages the lifecycle of application objects, centralising configuration and promoting loose coupling. This IoC container also supports dependency injection, simplifying the management of dependencies within the application.\nUse Cases # Tapestry is particularly well-suited for developers looking for a Java-based framework that supports rapid development cycles and needs a scalable solution for enterprise-grade applications. Its strong support for AJAX and client-side interactions makes it suitable for applications requiring rich, dynamic user interfaces.\nWhat is Apache Wicket? # Apache Wicket, often called Wicket, is a component-based web application framework for the Java programming language. It distinguishes itself with a strong emphasis on simplicity, ease of use, and a clean separation between logic and markup. Wicket allows developers to write applications in pure Java using components without the need for extensive XML configuration files.\nKey Features of Apache Wicket # Component-Based Architecture : Wicket is built around a component-based model that treats pages and their elements as reusable components. These components are backed by Java classes and associated HTML templates. This design enables the encapsulation of behaviour and promotes reuse.\nPure HTML Templates : Wicket uses pure HTML for its templates, allowing designers to create and edit them without needing to understand or embed any particular syntax or placeholders. Java components dynamically bind to these HTML templates via Wicket-specific attributes, keeping the development clean and straightforward.\nStrong Separation of Concerns : The framework promotes a clear separation between markup (HTML) and business logic (Java), making it easier for designers and developers to collaborate. Changes to the page layout do not typically require changes in the Java code.\nEvent-driven : Components in Wicket are event-driven, similar to Java Swing or AWT. They handle events through callback methods, simplifying the management of user interactions and partial page updates.\nStateful or Stateless : Wicket supports stateful components (preserving state across requests) and stateless components (where each request is treated as a new interaction), giving developers flexibility based on their application requirements.\nAJAX Support : The framework has built-in support for AJAX, allowing developers to update parts of a web page asynchronously without refreshing the entire page. This seamless integration, with Wicket managing much of the complexity behind the scenes.\nSecurity Features : Wicket is designed with security in mind, providing features to prevent common vulnerabilities such as Cross-Site Scripting (XSS) and Cross-Site Request Forgery (CSRF). These protections are built into the framework and enabled by default.\nTestability : The Wicket was designed to be testable at the unit and integration levels. It provides mock objects and testing mechanisms to simulate user interaction and application flow without requiring a running web server.\nUse Cases # Wicket is suitable for developers who prefer a pure Java approach to web development without relying heavily on JavaScript. It\u0026rsquo;s particularly appealing in environments where application maintainability and a clear separation between code and design are priorities. This makes it ideal for enterprise applications where long-term maintainability and scalability are critical.\nVaadin Plattform # The Vaadin Platform is a comprehensive development suite that simplifies building modern web applications. It is especially favoured for its ability to enable the creation of user interfaces (UIs) using Java, a language widely used across various industries, particularly in enterprise settings. Here are some key components and features of the Vaadin Platform:\nVaadin Flow : This is the core framework for building web UIs in Java. Flow lets developers manage the UI and its interactions on the server side, automating the communication between the client (browser) and server.\nVaadin Components : Vaadin comes with a rich set of built-in components, such as buttons, text fields, grids, and charts, ready to use out of the box. These components are built on web components, making them modern and highly interactive.\nVaadin Fusion : Targeted at developers who prefer client-side development, Fusion allows building web applications using TypeScript and leveraging the same backend APIs that Flow uses. This supports a more traditional approach to web development using JavaScript/TypeScript for the frontend while maintaining Java for the backend.\nThemes and Layouts : Vaadin supports customisable themes and layouts, allowing developers to tailor the appearance of their applications to match specific branding or aesthetic requirements. It uses standard CSS for styling.\nTools and Integrations : Vaadin provides various tools and integrations for popular IDEs (like Eclipse, IntelliJ IDEA, and VS Code), build tools (such as Maven and Gradle), and frameworks (like Spring). This integration helps streamline the development process, making it more efficient.\nEnd-to-End Development : From front-end components to back-end integration, Vaadin aims to provide a full-stack development experience. It supports building applications with responsive design, accessibility features, and internationalisation.\nCommunity and Pro Add-ons : The Vaadin platform also offers a range of community-driven and professional add-ons that extend the functionality of the basic framework. These can include advanced data components, tools for enhanced productivity, and integrations with other systems.\nVaadin Flow # Vaadin Flow is a Java framework for building modern web applications. Flow is specifically designed to let developers build web UIs in pure Java without needing HTML or JavaScript, though these can still be used if desired.\nHere\u0026rsquo;s how Vaadin Flow works:\nServer-Side Java : The application\u0026rsquo;s logic is written entirely in Java. Vaadin Flow automatically manages the communication between the client-side (browser) and the server-side.\nComponents and Layouts : Flow provides a range of UI components (like buttons, grids, and forms) and layouts that are used to construct the application interface.\nData Binding : It supports robust data binding capabilities, making it easy to connect UI components to data sources and synchronising data between the server and the client.\nRouting and Navigation : Vaadin Flow handles routing and navigation within the application, managing different views that users interact with.\nVaadin Flow is ideal for developers familiar with Java and prefer to work in a server-side programming environment. It allows them to create rich, interactive web applications while remaining trendy in enterprise environments where Java is a common choice for backend development.\nThe Difference between Apache Tapestry, Apache Wicket and Vaadin Flow # Apache Tapestry, Apache Wicket, and Vaadin Flow are all Java-based frameworks designed to facilitate the development of web applications through a component-oriented approach. However, they each have distinct characteristics, philosophies, and design implementations. Here\u0026rsquo;s a detailed comparison:\nApache Tapestry # Component Model : Tapestry uses a model where pages and components are Java objects that are mapped to templates defined in HTML files. Each component is highly reusable and can be embedded within other components.\nTemplates and Scripting : Tapestry firmly separates Java code from the HTML templates, promoting clean architecture and allowing designers to work on HTML/CSS without needing to understand Java.\nConventions over Configuration : Tapestry relies heavily on convention over configuration, reducing the need for XML configuration files and aiming for zero configuration where possible.\nLive Class Reloading : One of Tapestry\u0026rsquo;s hallmark features is its live class reloading capability, which allows developers to change code and see results immediately without restarting the server.\nPerformance : Tapestry includes built-in support for JavaScript and AJAX without requiring deep developer knowledge, automatically optimising client-server communication.\nApache Wicket # Component Model : Wicket treats web pages as a hierarchy of components, each backed by its own Java class and usually associated with an HTML template that defines its layout.\nState Management : Wicket is known for being a stateful framework by default, which means it automatically manages the state across requests, but it can also be configured to be stateless.\nHTML and Java Integration : Wicket allows dynamic content to be written in pure HTML, with placeholders linked to Java components. This enables a clean separation between the presentation layer and business logic.\nEvent Model : It follows an event-driven model similar to desktop GUI programming, making it intuitive for developers with experience in such environments.\nTestability : Wicket is designed to be highly testable with support for mocking and easy unit testing of individual components.\nVaadin Flow # Component Model : Vaadin Flow allows developers to build their UI entirely in Java or combine Java and templates. Vaadin 14 introduced LitTemplate, which lets developers define components using TypeScript and LitHTML.\nClient-Server Communication : Flow abstracts the client-server communication completely, allowing developers to focus on server-side Java without worrying about the frontend, which is automatically handled.\nRich Widget Set : Vaadin has a comprehensive set of highly interactive UI components that resemble desktop functionality.\nThemes and Layouts : It supports powerful theming capabilities with CSS and theming constructs that are easy to apply across different components.\nPerformance and Scalability : Vaadin manages client-server communication efficiently but can be challenged by the large amount of data sent over the network, especially in complex applications with numerous UI components.\nKey Differentiators # Approach to HTML/Java Separation : Tapestry and Wicket encourage more direct involvement with HTML, facilitating collaboration between designers and developers. Vaadin, in contrast, allows developers to work almost entirely in Java, which might streamline development but can distance the design process from HTML/CSS.\nState Management : Wicket is inherently stateful, which is beneficial for certain applications but can impact performance and scalability. Tapestry and Vaadin manage the state more transparently and can be more scalable in large applications.\nEase of Use and Learning Curve : Vaadin is often considered easier to learn for Java developers since it requires less HTML and JavaScript knowledge. Tapestry and Wicket have steeper learning curves but offer more control and flexibility over the UI.\nLive Reloading : Tapestry\u0026rsquo;s live class reloading is particularly useful for rapid development cycles, a feature not inherently present in Wicket or Vaadin.\nChoosing between these frameworks depends mainly on the project requirements, team skills, and specific needs regarding application scalability, maintainability, and development speed. Each framework has strengths and can be the best choice in different scenarios.\n","date":"25 September 2024","externalUrl":null,"permalink":"/posts/what-are-component-based-web-application-frameworks-for-java-developers/","section":"Posts","summary":"Tapestry, Wicket, and Vaadin # A component-oriented Java web application framework is a development framework that enables the construction of web applications in a modular way, using reusable, encapsulated components that manage their state, behaviour, and presentation. This approach allows developers to build complex user interfaces by assembling pre-built or custom components, like building blocks, each handling specific functionalities within the application.\n","title":"What are component-based web application frameworks for Java Developers?","type":"posts"},{"content":"","date":"25 September 2024","externalUrl":null,"permalink":"/tags/wicket/","section":"Tags","summary":"","title":"Wicket","type":"tags"},{"content":"","date":"12 September 2024","externalUrl":null,"permalink":"/tags/cwe-1123/","section":"Tags","summary":"","title":"CWE-1123","type":"tags"},{"content":"Self-modifying code refers to a type of code that alters its own instructions while it is executing. While this practice can offer certain advantages, such as optimisation and adaptability, it is generally discouraged due to the significant risks and challenges it introduces. For Java developers, using self-modifying code is particularly problematic because it undermines the codebase\u0026rsquo;s predictability, readability, and maintainability, and Java as a language does not natively support self-modification of its code.\nRisks Examples of Risky Practices Example Mitigation Strategies Example Mitigation Strategies Mitigation Strategies CVE-2014-0114 CVE-2013-2423 CVE-2015-1832 CVE-2012-0507 CVE-2019-12384 Mitigation Strategies Code Injection Attacks Remote Code Execution (RCE) Privilege Escalation Dynamic Code Loading Attacks Polymorphic Malware Evasion of Security Mechanisms Backdoors and Rootkits Tampering with Security Features Risks # Unpredictable Behavior: Self-modifying code can lead to unexpected program behaviour, making diagnosing and fixing bugs difficult.\nSecurity Vulnerabilities: Code that modifies itself can be a vector for various security attacks, including injection attacks and malware.\nMaintenance Difficulty: Such code is difficult to read and understand, making it more difficult to maintain and update.\nPerformance Issues: Self-modifying code can cause performance degradation due to the additional overhead of modifying and interpreting the changes at runtime.\nExamples of Risky Practices # Dynamic Class Loading: Java allows classes to be loaded at runtime using mechanisms such as reflection or custom class loaders. While dynamic class loading itself is not inherently wrong, using it excessively or without apparent necessity can lead to self-modifying behaviour.\nBytecode Manipulation: Using libraries like ASM or Javassist to modify Java bytecode at runtime can lead to self-modifying code. This practice is highly discouraged unless essential.\nReflection: While reflection is a powerful feature, it can be misused to modify private fields, methods, or classes, leading to behaviour that is hard to trace and debug.\nExample # An example of risky self-modifying behaviour in Java using bytecode manipulation:\nimport javassist.ClassPool; import javassist.CtClass; import javassist.CtMethod; public class SelfModifyingExample { public static void main(String[] args) { try { ClassPool pool = ClassPool.getDefault(); CtClass cc = pool.get(\u0026#34;TargetClass\u0026#34;); CtMethod m = cc.getDeclaredMethod(\u0026#34;targetMethod\u0026#34;); m.insertBefore(\u0026#34;{ System.out.println(\\\u0026#34;Method modified at runtime\\\u0026#34;); }\u0026#34;); cc.toClass(); TargetClass target = new TargetClass(); target.targetMethod(); } catch (Exception e) { e.printStackTrace(); } } } class TargetClass { public void targetMethod() { System.out.println(\u0026#34;Original method execution\u0026#34;); } } In this example, the TargetClass method targetMethod is modified at runtime to include an additional print statement. This kind of modification can lead to the aforementioned risks.\nMitigation Strategies # Avoid Runtime Code Modifications: Design your system in a way that minimises or eliminates the need for runtime code modifications.\nUse Design Patterns: Employ design patterns such as Strategy or State patterns that allow behaviour changes without altering the code at runtime.\nProper Use of Reflection: Use reflection sparingly and only when no other viable solution exists. Document its usage thoroughly.\nStatic Code Analysis: Use static code analysis tools to detect and prevent the introduction of self-modifying code.\nExcessive use of self-modifying code in Java is fraught with risks that can compromise your applications\u0026rsquo; security, maintainability, and performance. By adhering to best practices and using design patterns that promote flexibility and adaptability without modifying code at runtime, you can avoid the pitfalls associated with CWE-1123.\nA Reflection Example # Reflection in Java allows for introspection and manipulation of classes, fields, methods, and constructors at runtime. While powerful, excessive or improper use of reflection can lead to self-modifying behaviours, which aligns with CWE-1123. This can result in unpredictable behaviour, security vulnerabilities, and maintenance challenges.\nExample # Below is an example demonstrating the excessive use of reflection to modify a class\u0026rsquo;s behaviour at runtime, which can be considered a form of self-modifying code.\nimport java.lang.reflect.Field; import java.lang.reflect.Method; public class ReflectionExample { public static void main(String[] args) { try { // Original object creation MyClass original = new MyClass(); original.printMessage(); // Using reflection to modify the behavior at runtime Class\u0026lt;?\u0026gt; clazz = Class.forName(\u0026#34;MyClass\u0026#34;); Method method = clazz.getDeclaredMethod(\u0026#34;setMessage\u0026#34;, String.class); method.setAccessible(true); // Modify the private field value using reflection Field field = clazz.getDeclaredField(\u0026#34;message\u0026#34;); field.setAccessible(true); field.set(original, \u0026#34;Modified message\u0026#34;); // Verify the modification original.printMessage(); } catch (Exception e) { e.printStackTrace(); } } } class MyClass { private String message = \u0026#34;Original message\u0026#34;; public void printMessage() { System.out.println(message); } private void setMessage(String message) { this.message = message; } } In this example, the ReflectionExample class:\nCreates an instance of MyClass and prints the original message. It uses reflection to modify the private field message and the private method setMessage of MyClass. Changes the value of the message field and prints the modified message.\nThis example showcases how reflection can alter an object\u0026rsquo;s behaviour and state at runtime, leading to the issues outlined in CWE-1123.\nMitigation Strategies # Minimise Reflection Use: Avoid using reflection unless absolutely necessary. Prefer alternative design patterns that allow for flexibility without modifying the code at runtime.\nAccess Control: Ensure that fields and methods that should not be modified are kept private and final where possible to prevent unintended access.\nStatic Analysis Tools: Use static analysis tools to detect excessive use of reflection and other risky practices in the codebase.\nCode Reviews: Conduct thorough code reviews to identify and mitigate the use of self-modifying code through reflection.\nReflection is a powerful tool in Java, but misuse can lead to the risks associated with CWE-1123. By adhering to best practices and minimising the use of reflection to modify code at runtime, developers can maintain the security, predictability, and maintainability of their applications.\nExample of Dynamic Class Loading # Dynamic class loading in Java refers to the ability to load and unload classes at runtime. While this can be useful in specific scenarios, excessive or improper use can lead to self-modifying code behaviours, which align with CWE-1123. This can introduce risks such as unpredictable behaviour, security vulnerabilities, and maintenance challenges.\nBelow is an example demonstrating the excessive use of dynamic class loading to modify a class\u0026rsquo;s behaviour at runtime, which can be considered a form of self-modifying code.\npublic class DynamicClassLoadingExample { public static void main(String[] args) { try { // Load the original class ClassLoader classLoader = DynamicClassLoadingExample.class.getClassLoader(); Class\u0026lt;?\u0026gt; loadedClass = classLoader.loadClass(\u0026#34;MyClass\u0026#34;); // Create an instance of the loaded class Object instance = loadedClass.getDeclaredConstructor().newInstance(); // Invoke the original method loadedClass.getMethod(\u0026#34;printMessage\u0026#34;).invoke(instance); // Dynamically load the modified class classLoader = new CustomClassLoader(); loadedClass = classLoader.loadClass(\u0026#34;ModifiedClass\u0026#34;); // Create an instance of the modified class instance = loadedClass.getDeclaredConstructor().newInstance(); // Invoke the modified method loadedClass.getMethod(\u0026#34;printMessage\u0026#34;).invoke(instance); } catch (Exception e) { e.printStackTrace(); } } } // Original class definition class MyClass { public void printMessage() { System.out.println(\u0026#34;Original message\u0026#34;); } } // Custom class loader to simulate loading a modified class class CustomClassLoader extends ClassLoader { @Override public Class\u0026lt;?\u0026gt; loadClass(String name) throws ClassNotFoundException { if (\u0026#34;ModifiedClass\u0026#34;.equals(name)) { // Define a modified version of MyClass at runtime String modifiedClassName = \u0026#34;ModifiedClass\u0026#34;; String modifiedClassBody = \u0026#34;public class \u0026#34; + modifiedClassName + \u0026#34; {\u0026#34; + \u0026#34; public void printMessage() {\u0026#34; + \u0026#34; System.out.println(\\\u0026#34;Modified message\\\u0026#34;);\u0026#34; + \u0026#34; }\u0026#34; + \u0026#34;}\u0026#34;; byte[] classData = compileClass(modifiedClassName, modifiedClassBody); return defineClass(modifiedClassName, classData, 0, classData.length); } return super.loadClass(name); } private byte[] compileClass(String className, String classBody) { // Simulate compiling the class body into bytecode (in a real scenario, use a compiler API) // This is a placeholder for demonstration purposes return classBody.getBytes(); } } In this example, the DynamicClassLoadingExample class:\nLoads an original class MyClass and invokes its printMessage method. Dynamically loads a modified version of the class, ModifiedClass, using a custom class loader. Creates an instance of the modified class and invokes its printMessage method, which prints a different message.\nThis example showcases how dynamic class loading can alter a program\u0026rsquo;s behaviour at runtime, leading to the issues outlined in CWE-1123.\nMitigation Strategies # Avoid Unnecessary Dynamic Loading: Use dynamic class loading only when it is indispensable and cannot be avoided through other design patterns.\nSecure Class Loaders: Ensure custom class loaders are secure and do not load untrusted or malicious classes.\nStatic Analysis Tools: Use static analysis tools to detect excessive use of dynamic class loading and other risky practices in the codebase.\nCode Reviews: Conduct thorough code reviews to identify and mitigate the use of self-modifying code through dynamic class loading.\nJava-based CVEs based on CWE-1123 # While there might not be specific CVEs (Common Vulnerabilities and Exposures) explicitly labelled as being caused by CWE-1123 (Excessive Use of Self-Modifying Code), several Java-related vulnerabilities can arise from practices associated with self-modifying code. These typically involve dynamic class loading, reflection, and bytecode manipulation issues. Here are some examples of Java-based CVEs that relate to these practices:\nCVE-2014-0114 # Description: Apache Commons Collections Remote Code Execution Vulnerability\nIssue: This vulnerability involves using reflection to manipulate serialised data, leading to arbitrary code execution. It was found in the Apache Commons Collections library, where certain classes could be used to execute arbitrary code when deserialised. This is a form of self-modifying behaviour, as the serialised data could alter the program\u0026rsquo;s execution flow.\nImpact: Attackers could exploit this vulnerability to execute arbitrary commands on the server running the vulnerable application.\nCVE-2013-2423 # Description: Oracle Java SE Remote Code Execution Vulnerability\nIssue: This vulnerability arises from improper handling of certain methods in Java, leading to the execution of arbitrary code. It leverages reflection and class-loading mechanisms to inject and execute malicious code.\nImpact: Exploiting this vulnerability allows remote attackers to execute arbitrary code on the affected system, potentially leading to total system compromise.\nCVE-2015-1832 # Description: Android Remote Code Execution Vulnerability in Apache Cordova\nIssue: This vulnerability involves dynamic class loading and improper validation of inputs. It allowed attackers to inject malicious code into an Android application built with Apache Cordova by exploiting the WebView component.\nImpact: Successful exploitation could result in arbitrary code execution within the context of the affected application, leading to potential data leakage or further exploitation.\nCVE-2012-0507 # Description: Oracle Java SE Remote Code Execution Vulnerability\nIssue: This vulnerability involves using reflection and dynamic class loading to exploit a flaw in the Java Runtime Environment (JRE). The vulnerability allows an untrusted Java applet to break out of the Java sandbox and execute arbitrary code.\nImpact: Exploiting this vulnerability could allow an attacker to execute arbitrary code on the host system with the user\u0026rsquo;s privileges running the Java applet.\nCVE-2019-12384 # Description: FasterXML jackson-databind Deserialization Vulnerability\nIssue: This vulnerability involves the unsafe handling of deserialisation using the jackson-databind library. By exploiting polymorphic type handling, attackers could inject malicious code that gets executed during deserialisation.\nImpact: Successful exploitation could result in arbitrary code execution, leading to potential data breaches and system compromise.\nMitigation Strategies # Avoid Self-Modifying Code Practices: Do not use dynamic class loading, reflection, or bytecode manipulation unless absolutely necessary. When required, ensure proper validation and security measures are in place.\nUse Safe Deserialisation: Avoid deserialisation of untrusted data. If deserialisation is necessary, libraries and techniques that enforce strict type checking and validation should be used.\nApply Security Patches: Regularly update and patch libraries and frameworks to protect against known vulnerabilities.\nCode Reviews and Static Analysis: Conduct thorough code reviews and use static analysis tools to detect and mitigate the use of risky code practices.\nSecurity Best Practices: To reduce the attack surface, follow security best practices, such as least privilege, input validation, and secure coding guidelines.\nWhat kind of attacks or infection methods are based on CWE-1123? # CWE-1123 (Excessive Use of Self-Modifying Code) can lead to several types of attacks and infection methods due to such code\u0026rsquo;s unpredictable and dynamic nature. Here are some common attack vectors and infection methods associated with this vulnerability:\nCode Injection Attacks # Attackers exploit self-modifying code to inject malicious code into a program. This can occur through various means, such as manipulating input data that gets executed or modifying code at runtime to include harmful payloads.\nExample:\nSQL Injection: If an application dynamically constructs SQL queries using user input and modifies these queries at runtime, an attacker can inject malicious SQL commands to alter the behaviour of the database operations.\nRemote Code Execution (RCE) # Self-modifying code can enable Remote Code Execution by allowing attackers to modify or load classes and methods at runtime. This makes it easier to introduce and execute arbitrary code.\nExample:\nDeserialization Vulnerabilities: When an application deserialises data without proper validation, an attacker can inject objects that modify the code flow, leading to the execution of arbitrary code.\nPrivilege Escalation # Attackers can exploit self-modifying code to escalate their privileges within a system. They can bypass security checks and gain higher-level access by dynamically altering the code.\nExample:\nReflection Attacks: Using reflection, attackers can modify private fields and methods to escalate privileges, accessing parts of the system that would otherwise be restricted.\nDynamic Code Loading Attacks # Self-modifying code often involves dynamic loading of classes or bytecode manipulation, which can be exploited to load malicious code at runtime.\nExample:\nDynamic Class Loading: Attackers can trick the application into loading a malicious class that performs unwanted actions, such as exfiltrating data or modifying system files.\nPolymorphic Malware # Self-modifying code is commonly used in polymorphic malware, where the malware changes its code to evade detection by security software.\nExample:\nPolymorphic Virus: A virus that encrypts its payload and changes its decryption routine with each infection, making it difficult for antivirus programs to detect the malware\u0026rsquo;s signature.\nEvasion of Security Mechanisms # Self-modifying code can be used to evade security mechanisms such as firewalls, intrusion detection systems (IDS), and antivirus software by altering its code structure dynamically.\nExample:\nMetamorphic Malware: Similar to polymorphic malware, metamorphic malware reprograms itself completely with each infection, ensuring that no two copies of the malware are identical, thus evading signature-based detection.\nBackdoors and Rootkits # Attackers can use self-modifying code to install backdoors or rootkits that alter the behaviour of the operating system or application to provide persistent unauthorised access.\nExample:\nRootkits: A rootkit can use self-modifying code to hide its presence by altering kernel or application code to prevent detection by security tools.\nTampering with Security Features # Self-modifying code can be used to tamper with security features such as authentication mechanisms, encryption routines, and access controls.\nExample:\nTampering with Authentication: By dynamically modifying authentication checks, an attacker can bypass login mechanisms and gain unauthorised access to the system.\nBy understanding these attack vectors and implementing mitigation strategies, developers and security professionals can reduce the risks associated with self-modifying code and improve the overall security of their applications.\nHappy Coding\nSven\n","date":"12 September 2024","externalUrl":null,"permalink":"/posts/cwe-1123-excessive-use-of-self-modifying-code-for-java-developers/","section":"Posts","summary":"Self-modifying code refers to a type of code that alters its own instructions while it is executing. While this practice can offer certain advantages, such as optimisation and adaptability, it is generally discouraged due to the significant risks and challenges it introduces. For Java developers, using self-modifying code is particularly problematic because it undermines the codebase’s predictability, readability, and maintainability, and Java as a language does not natively support self-modification of its code.\n","title":"CWE-1123: Excessive Use of Self-Modifying Code for Java Developers","type":"posts"},{"content":"","date":"12 September 2024","externalUrl":null,"permalink":"/tags/remote-code-execution/","section":"Tags","summary":"","title":"Remote Code Execution","type":"tags"},{"content":"","date":"4 September 2024","externalUrl":null,"permalink":"/tags/iot/","section":"Tags","summary":"","title":"IoT","type":"tags"},{"content":" Introduction # TinkerForge is an innovative and modular hardware system that allows users to build electronic devices quickly and flexibly. Whether you\u0026rsquo;re an experienced engineer, a hobbyist, or a complete newbie, TinkerForge offers a range of components that can be easily connected and programmed, allowing for rapid prototyping and the creation of custom electronics projects. Since its launch, TinkerForge has gained popularity in various areas, including education, research, and industrial automation, due to its user-friendly design and extensive feature set.\nIntroduction The origins and evolution of TinkerForge TinkerForge architecture Bricks Bricklets Master-Extensions Programming and control TinkerForge-API TinkerForge-GUI Remote access and control Applications of TinkerForge Training Prototyping Industrial automation DIY projects and maker community Advantages of TinkerForge Limitations and challenges How do I start programming TinkerForge electronics with Java? Requirements TinkerForge architecture Connecting your TinkerForge hardware Connect Bricklets to the Master Brick Connect the master brick to the computer. Installing the TinkerForge Java API Writing the first Java program Running and testing the program Dealing with errors Event-driven programming Controlling actors Conclusion The origins and evolution of TinkerForge # TinkerForge was founded in 2011. The founders Matthias Bolte and Olaf Lüke imagined a system that would lower the hurdles in hardware development. Their goal was to provide a range of components that could be easily combined into complex systems without requiring in-depth knowledge of electronics or soldering skills.\nThe idea arose from the frustration the founders experienced while working on electronics projects, which often required a lot of time just to get the essential components to work together. They recognised that there was a need for a more intuitive and modular system that would allow anyone to create hardware projects quickly and effectively.\nOver the years, TinkerForge has grown from a niche product into a widely used platform supported by a vibrant community of developers and manufacturers. The system is now used in various applications, from simple DIY projects to advanced industrial automation systems.\nTinkerForge architecture # The TinkerForge system is based on a modular architecture consisting of three main component types: Bricks, Bricklets and Master Extensions.\nBricks # Bricks are the core building blocks of the TinkerForge system. They are small, self-contained modules that provide specific functions, such as controlling motors, reading sensors, or connecting to a computer. Each brick has a specific purpose and can be easily attached using a stackable system.\nThe most common types of bricks include:\nMaster Brick : The Master Brick is the central node that controls and supplies power to other Bricks and Bricklets. It provides the communication interface between the TinkerForge stack and a computer or microcontroller.\nESP32 Brick : The ESP32 Brick offers six Bricklet Connections and is equipped with a powerful ESP32 microcontroller.\nHAT-Brick : With the HAT Brick up to act, bricklets can be connected to a Raspberry Pi.\nIMU-Brick 2.0 : The Inertial Measurement Unit Brick contains a 9 DOF (Degrees of Freedom) sensor that provides orientation data, making it suitable for navigation and motion-tracking applications.\nBricklets # Bricklets are smaller modules that connect to bricks to expand their functionality. They are essentially sensors, actuators, or other peripheral devices that can be connected to a brick to provide additional input or output options. Bricklets are connected to Bricks via standardised connectors, and each can support multiple Bricklets.\nSome examples of Bricklets are:\nTemperature Bricklet : This module measures temperature and can be used in projects requiring environmental monitoring.\nDistance IR Bricklet : This sensor measures the distance to an object using infrared light, which is helpful in robotics or object detection applications.\nLCD 20x4 Bricklet : A simple display module that enables text output, perfect for building user interfaces or displaying sensor data.\nMaster-Extensions # Master Extensions are additional boards that can be connected to the master brick to provide additional functionality or expand connectivity. These extensions allow the TinkerForge system to connect to various communication protocols or external devices.\nExamples of master extensions are:\nWIFI extension 2.0 : This module adds wireless communication capabilities to the TinkerForge system, allowing bricks to be controlled remotely via a computer or smartphone.\nRS485 extension : This extension enables long-distance communication via RS485, making it suitable for industrial environments where long-distance wired connections are required.\nEthernet extension : This extension provides a wired Ethernet connection to the TinkerForge system, ensuring reliable and fast communication in networked environments.\nProgramming and control # One of TinkerForge\u0026rsquo;s outstanding features is its ease of programming. The system is designed to be accessible to users of all experience levels. TinkerForge offers various ways to control and program the Bricks and Bricklets, ensuring users can choose the best method.\nTinkerForge-API # TinkerForge provides a powerful and easy-to-use application programming interface (API) that allows users to control the Bricks and Bricklets from various programming languages ​​, including Python, C, C++, Java, Ruby and more. The API abstracts the complexity of hardware interaction and simplifies writing code that interacts with the various components.\nThe API is well-documented and includes examples for each supported language. This documentation provides detailed explanations of each feature, making it easier for users to understand how to control the hardware.\nhttps://www.tinkerforge.com/en/doc/Software/API_Bindings_Java.html\nTinkerForge-GUI # TinkerForge also offers a graphical user interface (GUI) called Brick Viewer for users who prefer a more visual approach. This software visually represents the connected Bricks and Bricklets and allows users to configure, monitor and control their devices without writing code.\nBrick Viewer is handy for beginners who are just starting with the TinkerForge system. It allows users to experiment with the hardware, view sensor data in real-time, and adjust settings without understanding the underlying code.\nhttps://www.tinkerforge.com/de/doc/Software/Brickv.html#brickv\nRemote access and control # TinkerForge also supports remote access and control, allowing users to interact with their devices over a network. The WIFI or Ethernet Master extensions allow users to control their TinkerForge systems over the Internet from anywhere in the world. This feature is precious for projects that require remote monitoring or control, such as home automation systems or remote sensing applications.\nApplications of TinkerForge # TinkerForge\u0026rsquo;s versatility has led to its use in various applications. The most common areas of use for TinkerForge include:\nTraining # TinkerForge is an excellent tool for electronics and programming lessons. Its modular design allows students to experiment with different components and see immediate results. The ease of use and extensive documentation make it accessible to learners of all ages.\nEducational institutions often use TinkerForge in robotics, embedded systems, and computer science courses. It allows students to create projects demonstrating key concepts such as sensor integration, motor control and data communication.\nPrototyping # One of TinkerForge\u0026rsquo;s strengths is its ability to enable rapid prototyping. Engineers and designers can quickly assemble and test their ideas without complex wiring or soldering. The system\u0026rsquo;s modular design means that components can be easily replaced or reconfigured as needed.\nTinkerForge is often used in the early stages of product development, where speed and flexibility are crucial. Using TinkerForge, developers can iterate their designs faster, reducing the time to market for new products.\nIndustrial automation # TinkerForge is also used in industrial automation applications where reliability and scalability are important. The system is suitable for use in industrial environments by supporting various communication protocols, such as RS485 and Ethernet.\nIn these environments, TinkerForge components can monitor and control machines, collect data from sensors, and automate processes. The modular design allows for easy expansion and customisation, making building complex systems tailored to specific requirements possible.\nDIY projects and maker community # TinkerForge has a strong following among DIYers and hobbyists. The system\u0026rsquo;s simplicity and flexibility make it ideal for creating custom electronics projects, such as home automation systems, robotics, and art installations.\nThe maker community around TinkerForge is active, and users share their projects, code and ideas online. This community-focused approach has led to the development of a wide range of open-source projects and resources, further increasing the value of the TinkerForge ecosystem.\nAdvantages of TinkerForge # TinkerForge offers several advantages that have contributed to its popularity:\nModularity : The ability to mix and match components allows users to create highly customised systems without designing everything from scratch.\nEase of use : With a straightforward API and visual interface, TinkerForge is accessible to users of all experience levels.\nScalability : TinkerForge can be used in small, simple projects and large, complex systems. Its modular design allows it to be easily expanded as needed.\nCross-platform support : TinkerForge\u0026rsquo;s API is available in multiple programming languages, making it easy to integrate into existing projects.\nCommunity and support : A strong community and comprehensive documentation ensure users have access to the resources they need to succeed.\nLimitations and challenges # Although TinkerForge is a powerful and flexible system, it also has its limitations:\nCosts : Although the price is reasonable, building complex systems with TinkerForge can be expensive, especially when multiple bricks and brackets are required.\nLimited computing power : Some bricks, especially the master brick, may not have the computing power required for demanding applications. In such cases, an external microcontroller or computer may be required.\nLearning curve : Although TinkerForge is designed to be user-friendly, there is still a learning curve, especially for those new to electronics or programming.\nDependence on proprietary hardware : TinkerForge components are proprietary, meaning users are somewhat locked into the ecosystem. While the open API and documentation help mitigate this, it is worth considering for long-term projects.\nTinkerForge represents a significant advancement in the field of modular electronics, providing a platform that bridges the gap between hardware and software. Its user-friendly design, modular architecture, and extensive documentation make it an ideal choice for a wide range of users.\nHow do I start programming TinkerForge electronics with Java? # Requirements # Before diving into Java programming with TinkerForge, you must have the following:\nBasic knowledge of Java : Familiarity with Java syntax and programming concepts.\nTinkerForge-Hardware : At least one Meister Brick and a Bricklet (e.g. Temperature Bricklet) for interaction.\nComputer with USB port : To connect the master brick to the computer.\nInternet connection : To download the required software and libraries.\nTinkerForge architecture # Before programming, it is essential to understand the vital components of the TinkerForge system:\nMaster Bricks : The central controller that connects to your computer via USB. It manages communication with all connected Bricklets.\nBricklets : Modular components that fulfil specific functions (e.g. sensors, actuators). They are connected to the master brick via standardised connectors.\nStackable System : Multiple bricks and bricklets can be stacked to create complex systems without soldering.\nConnecting your TinkerForge hardware # Connect Bricklets to the Master Brick # First, the Bricklets should be connected to the Master Brick. Care should be taken to ensure that the connection is tight and that the plugs\u0026rsquo; pins are aligned correctly.\nExample setup :\nMaster Brick: Connected via USB. Temperature Bricklet: Connected to one of the ports of the master brick. Connect the master brick to the computer. # After connecting the master brick to the computer using a USB cable, it should boot up, as indicated by the LEDs lighting up.\nInstalling the TinkerForge Java API # To interact with TinkerForge hardware via Java, you need the TinkerForge Java API. This example uses Maven to manage the dependencies. The coordinates for the TinkerForge dependencies are:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.tinkerforge\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;tinkerforge\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.1.33\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; Writing the first Java program # The following example shows a simple Java program that reads temperature data from the Temperature Bricklet. For this purpose, a Java class called “TemperatureReader ` created. The following imports are necessary for this program. Typically, the IDE should automatically display the relevant suggestions.\nimport com.tinkerforge.BrickletTemperature; import com.tinkerforge.IPConnection; import com.tinkerforge.TinkerforgeException; The temperature bracket\u0026rsquo;s UID is required below. The UID is usually printed on the bricklet or can be found via the TinkerForge Brick Viewer.\npublic class TemperatureReader { private static final String HOST = \u0026#34;localhost\u0026#34;; private static final int PORT = 4223; // Replace XYZ with your Bricklet UID private static final String TEMPERATURE_BRICKLET_UID = \u0026#34;XYZ\u0026#34;; } For simplicity, the actual program is implemented directly in the main method of the class.\npublic static void main(String[] args) { IPConnection ipcon = new IPConnection(); // Create IP connection // Create device object BrickletTemperature temp = new BrickletTemperature(TEMPERATURE_BRICKLET_UID, ipcon); try { ipcon.connect(HOST, PORT); // Connect to brickd System.out.println(\u0026#34;Connected to TinkerForge\u0026#34;); // Read temperature short temperature = temp.getTemperature(); System.out.println(\u0026#34;Temperature: \u0026#34; + temperature / 100.0 + \u0026#34; °C\u0026#34;); ipcon.disconnect(); } catch (TinkerforgeException e) { System.err.println(\u0026#34;Error: \u0026#34; + e.getMessage()); e.printStackTrace(); } } note : Replace “XYZ” with the actual UID of the temperature bricklet.\nIP connection : Manages the connection to the TinkerForge daemon (**brickd** ), which facilitates communication between the computer and the bricks/bricklets.\nBrickletTemperature : Represents the temperature bricklet element and allows interaction with its functions.\ngetTemperature(): This function gets the current temperature in hundredths of a degree Celsius. The value is converted to degrees Celsius by dividing by 100.0.\nRunning and testing the program # Start the TinkerForge daemon (“brickd”).\nBefore running the Java program, make sure the TinkerForge daemon is running. This daemon manages communication between the computer and the TinkerForge hardware.\nRunning the Java program\nAfter starting the method TemperatureReader.main() \u0026rsquo;\u0026rsquo; you can see the following output on the console.\nConnected to TinkerForge\nTemperature: 23,45 °C\nOf course, the temperature value varies depending on the environment.\nDealing with errors # If errors occur, note the following:\nIncorrect UID : Make sure the UID in the source code matches the UID of the Temperature Bricklet.\nDaemon is not running : Check if the **brickd** is running and connected to the master brick.\nConnection problems : Check the USB connections and ensure the master brick is properly connected to the computer.\nEvent-driven programming # Now that a simple Java program for reading temperature data has been created, advanced features can be explored. Instead of actively querying temperature data, callbacks can be set up to receive data asynchronously.\nimport com.tinkerforge.BrickletTemperature; import com.tinkerforge.IPConnection; import com.tinkerforge.TinkerforgeException; public class TemperatureCallback { private static final String HOST = \u0026#34;localhost\u0026#34;; private static final int PORT = 4223; private static final String TEMPERATURE_BRICKLET_UID = \u0026#34;XYZ\u0026#34;; // Replace XYZ with your Bricklet UID public static void main(String[] args) { IPConnection ipcon = new IPConnection(); // Create IP connection // Create device object BrickletTemperature temp = new BrickletTemperature(TEMPERATURE_BRICKLET_UID, ipcon); try { ipcon.connect(HOST, PORT); // Connect to brickd System.out.println(\u0026#34;Connected to TinkerForge\u0026#34;); // Register temperature callback temp.addTemperatureListener((temperature) -\u0026gt; { System.out.println(\u0026#34;Temperature Callback: \u0026#34; + temperature / 100.0 + \u0026#34; °C\u0026#34;); }); // Set callback period to 1 second (1000 ms) temp.setTemperatureCallbackPeriod(1000); // Keep the program running System.out.println(\u0026#34;Press Ctrl+C to exit.\u0026#34;); while (true) { Thread.sleep(1000); } } catch (TinkerforgeException | InterruptedException e) { System.err.println(\u0026#34;Error: \u0026#34; + e.getMessage()); e.printStackTrace(); } } } Explanation:\naddTemperatureListener : Registers a callback function that fires every time the temperature is updated.\nsetTemperatureCallbackPeriod : Defines the interval (in milliseconds) at which the callback is invoked.\nControlling actors # If actuators such as LEDs or motors have been connected via Bricklets, they can be controlled programmatically.\nExample: Controlling an LED bricklet\nimport com.tinkerforge.BrickletLED; import com.tinkerforge.IPConnection; import com.tinkerforge.TinkerforgeException; public class LEDControl { private static final String HOST = \u0026#34;localhost\u0026#34;; private static final int PORT = 4223; // Replace ABC with your LED Bricklet UID private static final String LED_BRICKLET_UID = \u0026#34;ABC\u0026#34;; public static void main(String[] args) { IPConnection ipcon = new IPConnection(); // Create IP connection BrickletLED led = new BrickletLED(LED_BRICKLET_UID, ipcon); // Create device object try { ipcon.connect(HOST, PORT); // Connect to brickd System.out.println(\u0026#34;Connected to TinkerForge\u0026#34;); // Turn LED on led.setColorLEDValue(\u0026#34;on\u0026#34;); System.out.println(\u0026#34;LED is ON\u0026#34;); // Wait for 2 seconds Thread.sleep(2000); // Turn LED off led.setColorLEDValue(\u0026#34;off\u0026#34;); System.out.println(\u0026#34;LED is OFF\u0026#34;); ipcon.disconnect(); } catch (TinkerforgeException | InterruptedException e) { System.err.println(\u0026#34;Error: \u0026#34; + e.getMessage()); e.printStackTrace(); } } } Conclusion # Programming with Java on the TinkerForge platform provides a powerful way to create interactive and intelligent electronic projects. There are hardly any limits to your imagination here. Thanks to the connection to Java, you can link your projects with many other technologies, which dramatically simplifies the integration of existing infrastructures and projects.\nHappy coding and creating!\n","date":"4 September 2024","externalUrl":null,"permalink":"/posts/iot-with-tinkerforge-and-java/","section":"Posts","summary":"Introduction # TinkerForge is an innovative and modular hardware system that allows users to build electronic devices quickly and flexibly. Whether you’re an experienced engineer, a hobbyist, or a complete newbie, TinkerForge offers a range of components that can be easily connected and programmed, allowing for rapid prototyping and the creation of custom electronics projects. Since its launch, TinkerForge has gained popularity in various areas, including education, research, and industrial automation, due to its user-friendly design and extensive feature set.\n","title":"IoT with TinkerForge and Java","type":"posts"},{"content":"","date":"4 September 2024","externalUrl":null,"permalink":"/categories/tinkerforge/","section":"Categories","summary":"","title":"TinkerForge","type":"categories"},{"content":"","date":"4 September 2024","externalUrl":null,"permalink":"/tags/tinkerforge/","section":"Tags","summary":"","title":"TinkerForge","type":"tags"},{"content":"In this part of the series about Vaadin Flow, I will show how I can create the basic framework for the graphic design of a work application. The focus here is on the design of the work area and the organisation of the individual logical application groups. In other words, we create the application layout that can be used for an industrial project.\nOf course, this also includes dealing with different languages ​​based on i18N.\nOne of the first steps in an application is often to think about how everything should be presented on the screen. If menus are used, headers and footers should be used for elements, and what about the basic design of the workspace? The questions here are to be understood as examples, as they will, of course, always depend on the context and the target group.\nAppLayout What is i18N - Internationalization in Java i18N in Vaadin Flow Conclusion AppLayout # AppLayout is a reasonably robust and easy-to-use solution that comes with the Vaadin Flow framework. This class provides the entry point for quickly building an application\u0026rsquo;s framework. We looked at this class briefly in the first part and will now go into more detail here.\nA classic Applayout consists of a header, where part of the navigation is usually housed. There is also a corresponding footer, which is usually used for lower-priority status messages. The middle is then divided into one to three parts. We will choose a classic two-column design here. The left side is the main navigation, the detailed navigation of which will then be visible in the header like a menu bar.\nThe layout consists of three sections: a horizontal navigation bar (called Navbar), a fold-out navigation bar (drawer), and a content area. An application\u0026rsquo;s main navigation blocks should be positioned in the navigation bar and/or drawer while rendering views in the content area.\nThe app layout is responsive and automatically adapts to the screen size of desktops, tablets, and mobile devices. This behaviour saves us a lot of customisation work that we would otherwise have to implement ourselves.\nThe navigation structure depends mainly on the number of elements to be shown and whether a sub-navigation is required.\nThe navigation bar is a good choice for multiple items (3-5) as they fit in the viewport without scrolling. If you need to display more items or small screen support is a priority, the Drawer is a better choice. It can accommodate a longer list of links without scrolling and collapses into a hamburger menu on small screens. Additionally, a vertical list of items is easier for the user to scan. If multi-level or hierarchical navigation is required, Draver is the first level. The secondary (and tertiary) navigation elements can be placed using the drawer or in the navigation bar (Navbar).\nThe implementation can then look like this.\nWe create the basic framework in MainLayout from the class AppLayout is derived. Here, we have to use the methods for setting the individual navigation areas. Let\u0026rsquo;s start by building the main navigation, which is on the left side in this example. Here you will see a combination of icon and descriptions for the main areas of the application. The necessary graphical elements are displayed along with the application title via the method addToDrawer(\u0026hellip;) set.\nThe application title is an instance of the class H1. Here the text to be displayed is simply selected using the method setText() handover.\nThe navigation elements are created using the class SideNavItem realised. The transfer parameters are a label. a target view and an icon. All SideNavItem - Instances are in the instance of the class SideNav summarised and transferred to the layout. In case the number of entries is larger and you want the user to be able to scroll through this list, the instance of the class SideNav is embedded into an instance of the class Scroller.\nIn the source code, it looks like this:\nprivate Scroller primaryNavigation() { SideNav sideNav = new SideNav(); sideNav.addItem( new SideNavItem(\u0026#34;Dashboard\u0026#34;, DashboardView.class, VaadinIcon.DASHBOARD.create()), new SideNavItem(\u0026#34;Orders\u0026#34;, AllOrdersView.class, VaadinIcon.CART.create()), new SideNavItem(\u0026#34;Customers\u0026#34;, CustomersView.class, VaadinIcon.USER_HEART.create()), new SideNavItem(\u0026#34;Products\u0026#34;, ProductsView.class, VaadinIcon.PACKAGE.create()), new SideNavItem(\u0026#34;Documents\u0026#34;, DocumentsView.class, VaadinIcon.RECORDS.create()), new SideNavItem(\u0026#34;Tasks\u0026#34;, TasksView.class, VaadinIcon.LIST.create()), new SideNavItem(\u0026#34;Analytics\u0026#34;, AnalyticsView.class, VaadinIcon.CHART.create())); Scroller scroller = new Scroller(pageNav); scroller.setClassName(LumoUtility.Padding.SMALL); return scroller; } To complete the layout, we will put all the elements together.\nThis is implemented in the constructor.\nprivate H1 appTitle = new H1(); public MainLayout() { addToDrawer(appTitle, primaryNavigation()); setPrimarySection(Section.DRAWER); } In this example, you can see that the labels are all stored directly in the source code. We will change this later so that we can make the application multilingual. I ask for a bit of patience here. We also see the specification of the targets using the class name. We\u0026rsquo;ll look at that a little later, too.\nIn the previous post, I showed how the MainLayout class is used.\nScreen with the created menu items\nIf you want to create a sub-navigation, you can make additional menu entries in the header on the right. This is often used when you want to divide a somewhat more complex process into different views. For example, let\u0026rsquo;s take the illustration for dealing with orders. Let’s just assume that there are the sub-areas “All orders ”, “Open orders ”, “Completed orders \u0026quot; and \u0026ldquo;Abandoned orders ” must give. This results in another four navigation destinations, which are, however, summarised under the item Orders.\nScreenshot after navigating to the Orders menu item\nThese submenu items are the same for all views that deal with orders. This immediately raises the question of how to extract this menu. Great mechanisms from inheritance come into play here. We create a class AbstractHeaderView and AbstractOrdersView. All constructs required for a generic implementation of secondary navigation are stored in the first class mentioned. This includes, for example, how the second navigation bar should be positioned, what graphic properties it has and whether there is a hamburger symbol that can be used to show or hide the main navigation.\nUltimately, all that needs to be added is which navigation points must be created for each implementation. The respective implementation of the method secondaryNavigationsLinks() defines these menu items.\nThe following listing shows the implementation for the screenshot shown, divided into the generic part AbstractHeaderView and the specific one for the orders AbstractOrdersView.\npublic abstract class AbstractViewHeader extends Composite\u0026lt;VerticalLayout\u0026gt; { public AbstractViewHeader(String subTitle) { initView(subTitle); } protected RouterLink createLink(String viewName, Class\u0026lt;? extends Composite\u0026gt; viewClass) { RouterLink link = new RouterLink(); link.add(viewName); link.setRoute(viewClass); link.addClassNames(LumoUtility.Display.FLEX, LumoUtility.AlignItems.CENTER, LumoUtility.Padding.Horizontal.MEDIUM, LumoUtility.TextColor.SECONDARY, LumoUtility.FontWeight.MEDIUM); link.getStyle().set(\u0026#34;text-decoration\u0026#34;, \u0026#34;none\u0026#34;); return link; } public void initView(String subTitle) { DrawerToggle toggle = new DrawerToggle(); H2 viewTitle = new H2(getTranslation(subTitle)); HorizontalLayout titleBar = new HorizontalLayout(toggle, viewTitle); titleBar.setAlignItems(FlexComponent.Alignment.CENTER); titleBar.setSpacing(false); VerticalLayout viewHeader = getContent(); viewHeader.add(titleBar, secondaryNavigation()); viewHeader.setPadding(false); viewHeader.setSpacing(false); } private HorizontalLayout secondaryNavigation() { HorizontalLayout navigation = new HorizontalLayout(); navigation.addClassNames( LumoUtility.JustifyContent.CENTER, LumoUtility.Gap.SMALL, LumoUtility.Height.MEDIUM); secondaryNavigationLinks().forEach(navigation::add); return navigation; } public abstract List\u0026lt;RouterLink\u0026gt; secondaryNavigationLinks(); public abstract class AbstractOrdersView extends AbstractView\u0026lt;HorizontalLayout\u0026gt; implements LocaleChangeObserver { public static final String CANCELLED = \u0026#34;orders.subnavigation.cancelled\u0026#34;; public static final String COMPLETED = \u0026#34;orders.subnavigation.completed\u0026#34;; public static final String OPEN = \u0026#34;orders.subnavigation.open\u0026#34;; public static final String ALL = \u0026#34;orders.subnavigation.all\u0026#34;; private final String subTitle; public AbstractOrdersView(String subTitle) { this.subTitle = subTitle; } @Override protected AbstractViewHeader createViewHeader() { return new AbstractViewHeader(subTitle) { @Override public List\u0026lt;RouterLink\u0026gt; secondaryNavigationLinks() { return List.of( createLink(getTranslation(ALL), AllOrdersView.class), createLink(getTranslation(OPEN), OpenOrdersView.class), createLink(getTranslation(COMPLETED), CompletedOrdersView.class), createLink(getTranslation(CANCELLED), CancelledOrdersView.class)); } }; } } Here, you can clearly see that the source text is greatly reduced when there are a large number of navigation elements. A minimal basic framework is created to implement the variations for the individual views. I have intentionally refrained from including functions for processing orders in this example because I want to focus on the structure here.\n@Route(value = \u0026#34;allOrders\u0026#34;, layout = MainLayout.class) @PreserveOnRefresh public class AllOrdersView extends AbstractOrdersView { public static final String SUB_TITLE = \u0026#34;orders.all.subtitle\u0026#34;; public AllOrdersView() { super(SUB_TITLE); //the necessary elements are placed here getContent().add(new TextField(\u0026#34;TBD ALL Orders\u0026#34;)); } @Override public void localeChange(LocaleChangeEvent event) { // fuer i18N } } With the basic framework shown here, you can build quite complex navigations. Now, let\u0026rsquo;s get to the topic of i18n. A web application usually needs to support multiple languages. I will not discuss languages ​​here that require a change in reading direction. In this example, I\u0026rsquo;ll focus on supporting any number of languages ​​with a left-to-right reading order. Examples of languages ​​here will be German and English. The primary function here is quite simple\nWhat is i18N - Internationalization in Java # The basic concept of i18n (internationalisation) in Java is to design and develop an application that can be easily adapted to different languages ​​and regions without requiring significant changes to the code.\nResource bundle\nA resource bundle is a set of key-value pairs that store country-specific data, such as B. Strings for messages, labels and other UI components. In Java, resource bundles are usually property files named after the locale they represent (e.g. \u0026ldquo;messages_en_US.properties \u0026quot; for US English, \u0026ldquo;messages_de.properties “ for general German).\nWhen the application runs, it dynamically loads the appropriate resource bundle based on the user\u0026rsquo;s locale. This allows the application to display content in the user\u0026rsquo;s preferred language without hard-coding the text in the source code.\nLocale\nA locale in Java is an object that represents a specific geographical, political or cultural region. It includes language, country and sometimes variant codes. For example, Locale.ENGLISH represents the English language and Locale.US represents the United States.\nThe class **Local** customises information for the user, such as B. Text, dates, numbers and currencies, according to its region.\nMessage Formatting\nJava provides classes like **MessageFormat** to handle complex message formatting that allows dynamic insertion of values ​​into localised messages (e.g. \u0026ldquo;Welcome, {0}!\u0026rdquo;, where {0} can be replaced with the user\u0026rsquo;s name).\nYou define message templates in your resource bundles, and at runtime, the class replaces **MessageFormat** Placeholders with actual values ​​to ensure messages are appropriately localised and formatted.\nDate, number and currency formatting\nDates, numbers, and currencies are often represented differently in different locales (e.g. MM/DD/YYYY vs. DD/MM/YYYY for dates). Java provides classes like **DateFormat** , **NumberFormat** and **Currency** to process these deviations.\nThese classes format data according to locale settings and ensure users see information in the expected format.\nFallback mechanism\nIf a specific locale\u0026rsquo;s resource package is unavailable, Java uses a fallback mechanism to select a more general package (e.g., falling back from messages_fr_CA.properties to messages_fr.properties).\nThis ensures that the application remains functional even if a full set of localised resources is unavailable for each locale.\nThe basic concept of i18n in Java is to create applications that can support multiple languages ​​and regional settings with minimal changes to the code base. This includes using resource bundles, locale-dependent classes for formatting, and a fallback mechanism to ensure a seamless user experience across different locales.\ni18N in Vaadin Flow # To use i18N, the application only needs to have the corresponding text files with the keys and contents available in the classpath under the vaadin-i18n directory with the filename prefix translations (e.g. src/main/resources/vaadin-i18n/translations.properties).\nWhen a translation is requested or when the I18NProvider is used for the first time, the vaadin-i18n directory is checked to see if it contains any translations.properties or translations_[langcode].properties files. All language codes are collected from the available files and added as provided locales in the DefaultI18NProvider.\nThe translations.properties file is a standard translation file used for any locale that does not have a specific translation file. For example, locale translation files are called translations_de.properties or translations_en.properties. Automatic locale creation supports one to three parts (e.g. translations_language_country_variant.properties).\nThe locale to use is determined by matching the locales provided by the I18NProvider with the Accept-Language header in the client\u0026rsquo;s initial response. It will be used if an exact match (i.e., language and country) is found. Otherwise, it will just try to find a match by language. If neither is found, the locale defaults to the first \u0026ldquo;supported\u0026rdquo; locale from I18NProvider.getProvidedLocales() set. If this is empty, will Locale.getDefault() used.\nImplementing internationalisation in an application is a combination of using I18NProvider and updating translations when locale changes. This can be done, for example, by the user if he wants to change the language. To simplify this, the application classes that control the localised captions and text can LocaleChangeObserver implement to receive events related to locale changes. During navigation, this observer will also be notified when the component is attached, but before onAttach() is called. All URL parameters from navigation are set so that they can be used to determine status.\npublic class LocaleObserver extends Div implements LocaleChangeObserver { @Override public void localeChange(LocaleChangeEvent event) { setText(getTranslation(\u0026#34;my.translation\u0026#34;, getUserId())); } } If you want to set the texts explicitly, you can use this method: getTranslation(..).\npublic class MyLocale extends Div { public MyLocale() { setText(getTranslation(\u0026#34;my.translation\u0026#34;, getUserId())); } } In this example project, the listing for creating the main menu entries is rewritten. For clarity, here are both versions for a menu entry.\nHigh Version :\nnew SideNavItem(\u0026#34;Orders\u0026#34;, AllOrdersView.class, VaadinIcon.CART.create()) New version :\npublic static final String MENU_ITEM_ORDERS = \u0026#34;mainlayout.menuitem.orders\u0026#34;; new SideNavItem( getTranslation(MENU_ITEM_ORDERS), AllOrdersView.class, CART.create()) translation.properties :\nmainlayout.menuitem.orders=Orders\nConclusion # With these two elements, the ApplLayout and the realisation or implementation of i18N, two fundamental building blocks for developing a web application have been laid. From here, the application can now be built step by step. We will deal with the individual topics in the next parts, so it remains exciting.\nHappy Coding\nSven\n","date":"22 August 2024","externalUrl":null,"permalink":"/posts/building-more-complex-apps-with-flow/","section":"Posts","summary":"In this part of the series about Vaadin Flow, I will show how I can create the basic framework for the graphic design of a work application. The focus here is on the design of the work area and the organisation of the individual logical application groups. In other words, we create the application layout that can be used for an industrial project.\n","title":"Building More Complex Apps with Flow","type":"posts"},{"content":"In software development, temporary files are often used to store data temporarily during an application’s execution. These files may contain sensitive information or be used to hold data that must be processed or passed between different parts of a program. However, if these temporary files are not managed securely, they can introduce vulnerabilities that may compromise the application\u0026rsquo;s confidentiality, integrity, or availability. The Common Weakness Enumeration (CWE) identified CWE-377 as a weakness associated with the insecure creation and management of temporary files.\nUnderstanding CWE-377 CWE-377 in Java Example of Vulnerable Code Potential Impacts Mitigations Use File.createTempFile() Properly Ensure Proper File Permissions Avoid Hardcoded File Names Handle Temporary Files in a Privileged Context Advanced Considerations Use java.nio.file Package Consider Using In-Memory Solutions Best Practices OpenSource Libraries for secure temp file handling Apache Commons IO Google Guava JUnit 5 (JUnit Jupiter) Practical examples RESTful API Demo - Upload with Javalin Insecure Implementation Secure Implementation Key Differences Conclusion on CWE-377: Insecure Temporary File Understanding CWE-377 # CWE-377: Insecure Temporary File refers to a security weakness that occurs when a program creates a temporary file in an insecure manner. Attackers can exploit this vulnerability to perform various malicious activities, including data tampering, unauthorised data access, or denial of service. Insecure temporary file creation typically involves issues such as:\nPredictable file names : If temporary files have predictable names, attackers can guess the file names and either read or modify the file contents.\nInsecure file permissions : Incorrect permissions can allow unauthorised users to access or modify temporary files.\nRace conditions : A time-of-check to time-of-use (TOCTOU) race condition may occur if an attacker creates a file with the same name as the one the application intends to generate before the application does.\nCWE-377 in Java # In Java, developers often use temporary files for various purposes, such as caching, processing intermediate data, or storing temporary results. Java provides several ways to create temporary files, such as the **File.createTempFile()** method, which generates a temporary file with a unique name in the default temporary-file directory. However, improper use of these APIs can lead to CWE-377.\nExample of Vulnerable Code # Consider the following example:\nimport java.io.File; import java.io.IOException; public class InsecureTempFileExample { public static void main(String[] args) throws IOException { File tempFile = new File(\u0026#34;/tmp/tempfile.txt\u0026#34;); tempFile.createNewFile(); System.out.println(\u0026#34;Temporary file created at: \u0026#34; + tempFile.getAbsolutePath()); } } In this example, the code creates a temporary file with a hardcoded file name (tempfile.txt) in the /tmp directory. This approach is insecure for several reasons: The file name is predictable, allowing an attacker to create a file with the same name before the application does, leading to a TOCTOU race condition. The file is created without any control over file permissions, potentially exposing sensitive data.\nPotential Impacts # CWE-377 can have severe consequences depending on how the temporary file is used and the sensitivity of the data it contains. Some potential impacts include:\nInformation Disclosure : If an attacker can predict the name of a temporary file, they may be able to read its contents if the file is not adequately protected. This can lead to disclosing sensitive information, such as passwords, tokens, or personal data.\nData Tampering : An attacker could create or modify a temporary file before the application uses it, resulting in data corruption or unauthorised modifications. This could lead to incorrect application behaviour or the introduction of malicious data into the system.\nDenial of Service (DoS) : By preemptively creating temporary files with the names an application expects to use, an attacker can prevent the application from functioning correctly, leading to a denial of service.\nMitigations # To prevent CWE-377, developers must adhere to secure coding practices when working with temporary files. Below are several mitigation strategies for securely creating and managing temporary files in Java:\nUse File.createTempFile() Properly # The File.createTempFile() method generates a unique temporary file name, reducing the risk of predictable file names. It also allows developers to specify a directory for the file, though it defaults to the system\u0026rsquo;s temporary-file directory.\nimport java.io.File; import java.io.IOException; public class SecureTempFileExample { public static void main(String[] args) throws IOException { File tempFile = File.createTempFile(\u0026#34;tempfile_\u0026#34;, \u0026#34;.tmp\u0026#34;); tempFile.deleteOnExit(); // Ensures the file is deleted when the JVM exits System.out.println(\u0026#34;Temporary file created at: \u0026#34; + tempFile.getAbsolutePath()); } } This approach mitigates several risks:\nThe file name is generated randomly, making it difficult for an attacker to predict. The file is automatically deleted when the Java Virtual Machine (JVM) exits, reducing the likelihood of stale files. Ensure Proper File Permissions # When creating temporary files, setting appropriate file permissions is crucial to prevent unauthorised access. In Java, you can use the **setReadable()** , **setWritable()** , and **setExecutable()** methods to control file permissions.\nimport java.io.File; import java.io.IOException; public class SecureTempFileWithPermissionsExample { public static void main(String[] args) throws IOException { File tempFile = File.createTempFile(\u0026#34;secure_tempfile_\u0026#34;, \u0026#34;.tmp\u0026#34;); tempFile.setReadable(true, true); tempFile.setWritable(true, true); tempFile.setExecutable(false); tempFile.deleteOnExit(); System.out.println(\u0026#34;Temporary file created with secure permissions at: \u0026#34; + tempFile.getAbsolutePath()); } } In this example, the file is readable and writable only by the owner, minimising the risk of unauthorised access.\nAvoid Hardcoded File Names # Using hardcoded file names, as seen in the initial insecure example, is risky because it makes the temporary file name predictable. Always use mechanisms that generate unique, unpredictable file names, such as **File.createTempFile()** .\nHandle Temporary Files in a Privileged Context # Managing temporary files in a more controlled or privileged environment may be beneficial when dealing with sensitive data. This could involve creating the files in a directory with restricted access or using Java’s **AccessController** to enforce stricter security policies around file operations.\nAdvanced Considerations # Use java.nio.file Package # The **java.nio.file** package, introduced in Java 7, provides more robust and flexible mechanisms for file handling, including temporary file creation. The **Files** class offers the **createTempFile()** method, which can also specify file attributes like permissions.\nimport java.nio.file.Files; import java.nio.file.Path; import java.nio.file.attribute.PosixFilePermissions; import java.util.Set; public class NioSecureTempFileExample { public static void main(String[] args) throws IOException { Set\u0026lt;PosixFilePermission\u0026gt; permissions = PosixFilePermissions.fromString(\u0026#34;rw-------\u0026#34;); Path tempFile = Files.createTempFile(\u0026#34;nio_tempfile_\u0026#34;, \u0026#34;.tmp\u0026#34;, PosixFilePermissions.asFileAttribute(permissions)); System.out.println(\u0026#34;Temporary file created with NIO at: \u0026#34; + tempFile.toAbsolutePath()); } } This example shows how to set file permissions using the NIO package, providing fine-grained control over file security attributes.\nConsider Using In-Memory Solutions # Some applications may be able to avoid creating temporary files altogether by using in-memory storage solutions, such as **ByteArrayOutputStream** , for temporary data. This approach eliminates the risks associated with file-system-based temporary storage but may not be suitable for large data sets or applications with significant memory constraints.\nBest Practices # To ensure secure temporary file handling in Java, developers should adopt the following best practices:\nPrefer Secure Defaults : Always use methods like **File.createTempFile()** or **Files.createTempFile()** that provide secure defaults for file creation.\nSet File Permissions Explicitly : Ensure that temporary files have the minimal necessary permissions and avoid granting unnecessary access to other users.\nAvoid Predictable File Names : Never use hardcoded or predictable names for temporary files. Always generate unique file names using secure APIs.\nUse deleteOnExit() : When possible, use **deleteOnExit()** to ensure that temporary files are cleaned up automatically when the JVM terminates.\nLimit Scope of Temporary Files : Store temporary files in directories with restricted access, and consider creating a dedicated directory for temporary files that require stricter security controls.\nHandle Exceptions Gracefully : Always handle exceptions when working with temporary files, ensuring the application remains secure and stable even if file operations fail.\nCWE-377: Insecure Temporary File is a significant security concern in software development, particularly in sensitive data processing environments. In Java, developers must be vigilant in creating, managing, and securing temporary files to avoid introducing vulnerabilities that malicious actors could exploit.\nBy following the guidelines and best practices outlined in this document, Java developers can mitigate the risks associated with CWE-377 and ensure that their applications handle temporary files securely. The use of secure APIs, proper file permissions, and cautious file management practices are essential for maintaining the confidentiality, integrity, and availability of data within a Java application.\nEnsuring that temporary files are managed securely is not only a best practice but a critical aspect of developing robust, secure Java applications. By adhering to these principles, developers can protect their applications from common threats and contribute to a safer software ecosystem.\nOpenSource Libraries for secure temp file handling # Regarding secure temporary file handling in Java, several open-source libraries can help simplify and enhance security for managing temporary files. Here are some notable ones:\nApache Commons IO # Apache Commons IO provides a utility class for securely creating and managing temporary files.\nMaven Dependency :\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;commons-io\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;commons-io\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.11.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; Example Usage :\nimport org.apache.commons.io.FileUtils; import java.io.File; import java.io.IOException; public class SecureTempFileExample { public static void main(String[] args) { try { // Create a secure temporary file File tempFile = FileUtils.getTempDirectory() .createTempFile(\u0026#34;secureTempFile\u0026#34;, \u0026#34;.txt\u0026#34;); tempFile.deleteOnExit(); // Ensure the file is deleted on exit // Optional: Set specific file permissions (POSIX systems) if (tempFile.exists()) { tempFile.setReadable(true, true); tempFile.setWritable(true, true); tempFile.setExecutable(false, false); } System.out.println(\u0026#34;Temporary file created at: \u0026#34; + tempFile.getAbsolutePath()); } catch (IOException e) { e.printStackTrace(); } } } Google Guava # Google Guava also provides utility methods for working with temporary files.\nMaven Dependency :\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.google.guava\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;guava\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;31.0.1-jre\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; Example Usage :\nimport com.google.common.io.Files; import java.io.File; import java.io.IOException; public class SecureTempFileExample { public static void main(String[] args) { try { // Create a secure temporary file File tempFile = Files.createTempDir(); tempFile.deleteOnExit(); // Ensure the file is deleted on exit // Optional: Set specific file permissions (POSIX systems) if (tempFile.exists()) { tempFile.setReadable(true, true); tempFile.setWritable(true, true); tempFile.setExecutable(false, false); } System.out.println(\u0026#34;Temporary file created at: \u0026#34; + tempFile.getAbsolutePath()); } catch (Exception e) { e.printStackTrace(); } } } JUnit 5 (JUnit Jupiter) # JUnit 5 provides a TempDir extension for securely creating temporary directories and files for testing purposes.\nMaven Dependency :\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.junit.jupiter\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;junit-jupiter-engine\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;5.8.2\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; Example Usage :\nimport org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; import java.io.File; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; public class SecureTempFileTest { @TempDir Path tempDir; @Test public void testCreateSecureTempFile() throws IOException { // Create a secure temporary file in the provided temp directory Path tempFile = Files.createTempFile(tempDir, \u0026#34;secureTempFile\u0026#34;, \u0026#34;.txt\u0026#34;); // Optional: Set specific file permissions (POSIX systems) Set\u0026lt;PosixFilePermission\u0026gt; permissions = PosixFilePermissions.fromString(\u0026#34;rw-------\u0026#34;); Files.setPosixFilePermissions(tempFile, permissions); System.out.println(\u0026#34;Temporary file created at: \u0026#34; + tempFile.toAbsolutePath()); } } Using these libraries can significantly simplify secure temporary file handling in Java applications. Apache Commons IO and Google Guava provide utilities for creating and managing temporary files, while JUnit 5 offers built-in support for creating temporary files and directories during testing.\nChoose the library that best fits your needs and project dependencies. Apache Commons IO and Google Guava are versatile and widely used, whereas JUnit 5\u0026rsquo;s TempDir is excellent for secure temporary file handling in test cases.\nLet\u0026rsquo;s discover some examples.\nPractical examples # RESTful API Demo - Upload with Javalin # Javalin is a lightweight web framework for Java and Kotlin that can be used to create RESTful APIs and web applications. Below, I will provide two examples of handling file uploads and creating temporary files in Javalin: insecure and secure.\nInsecure Implementation # In this insecure example, we will create temporary files with predictable names without setting appropriate permissions.\nimport io.javalin.Javalin; import java.io.File; import java.io.IOException; public class InsecureTempFileExample { public static void main(String[] args) { Javalin app = Javalin.create().start(7000); app.post(\u0026#34;/upload\u0026#34;, ctx -\u0026gt; { // Save uploaded file to a temporary file // with a predictable name File uploadedFile = ctx.uploadedFile(\u0026#34;file\u0026#34;).getContent(); File tempFile = new File(\u0026#34;/tmp/insecure-temp-file.txt\u0026#34;); try { uploadedFile.renameTo(tempFile); ctx.result(\u0026#34;File uploaded to: \u0026#34; + tempFile.getAbsolutePath()); } catch (Exception e) { ctx.result(\u0026#34;File upload failed\u0026#34;); e.printStackTrace(); } }); } } Secure Implementation # In this secure example, we will use Files.createTempFile to create temporary files with unique, unpredictable names and set appropriate file permissions.\nimport io.javalin.Javalin; import java.io.InputStream; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.attribute.PosixFilePermission; import java.nio.file.attribute.PosixFilePermissions; import java.util.Set; public class SecureTempFileExample { public static void main(String[] args) { Javalin app = Javalin.create().start(7000); app.post(\u0026#34;/upload\u0026#34;, ctx -\u0026gt; { // Retrieve uploaded file InputStream uploadedFile = ctx.uploadedFile(\u0026#34;file\u0026#34;).getContent(); // Define the prefix and suffix for the temporary file String prefix = \u0026#34;secureTempFile\u0026#34;; String suffix = \u0026#34;.txt\u0026#34;; try { // Create temporary file with default permissions Path tempFile = Files.createTempFile(prefix, suffix); // Optional: Set specific file permissions (POSIX systems) Set\u0026lt;PosixFilePermission\u0026gt; permissions = PosixFilePermissions.fromString(\u0026#34;rw-------\u0026#34;); Files.setPosixFilePermissions(tempFile, permissions); // Write uploaded file content to temporary file Files.copy(uploadedFile, tempFile); ctx.result(\u0026#34;File uploaded to: \u0026#34; + tempFile.toAbsolutePath().toString()); } catch (IOException e) { ctx.result(\u0026#34;File upload failed\u0026#34;); e.printStackTrace(); } }); } } Key Differences # File Creation : * **Insecure** : Uses a predictable file name (/tmp/insecure-temp-file.txt), which can lead to filename collision and unauthorised access. * **Secure** : Uses Files.createTempFile to create a unique, unpredictable file name. File Permissions : * **Insecure** : Does not explicitly set file permissions, which may result in insecure default permissions. * **Secure** : Explicitly sets restrictive file permissions (rw-------), ensuring only the owner can read and write the file. Exception Handling : * **Both examples** : Handle exceptions, but the secure example properly handles IOException that may arise during file operations. By following the secure example, you can mitigate the risks associated with CWE-377 and ensure that temporary files are created securely in your Javalin applications.\nConclusion on CWE-377: Insecure Temporary File # CWE-377, concerning the insecure creation and management of temporary files, is a critical security vulnerability that can have far-reaching consequences if not adequately addressed. In the context of Java, this weakness often arises due to predictable file names, inadequate file permissions, and race conditions such as Time-of-Check to Time-of-Use (TOCTOU). These issues can lead to severe threats, including unauthorised access, data tampering, information leakage, and even denial of service.\nTo mitigate the risks associated with CWE-377, developers must adopt secure practices when handling temporary files. This includes using secure and atomic file creation methods, like **File.createTempFile()** and **Files.createTempFile()** , which generate unique and unpredictable file names while ensuring the atomicity of operations. Proper file permissions must also be enforced to limit access to temporary files, preventing unauthorised users from reading or modifying them.\nUnderstanding the nuances of race conditions like TOCTOU is essential in designing secure applications. By eliminating the gap between checking a file’s existence and subsequent use, developers can prevent attackers from exploiting the system during that critical window.\nIn summary, addressing CWE-377 requires a combination of secure coding practices, careful file permissions management, and robust APIs that inherently mitigate common pitfalls associated with temporary files. By adhering to these practices, Java developers can protect their applications from the potentially severe impacts of insecure temporary file management, ultimately ensuring their software systems\u0026rsquo; confidentiality, integrity, and availability.\n","date":"21 August 2024","externalUrl":null,"permalink":"/posts/cwe-377-insecure-temporary-file-in-java/","section":"Posts","summary":"In software development, temporary files are often used to store data temporarily during an application’s execution. These files may contain sensitive information or be used to hold data that must be processed or passed between different parts of a program. However, if these temporary files are not managed securely, they can introduce vulnerabilities that may compromise the application’s confidentiality, integrity, or availability. The Common Weakness Enumeration (CWE) identified CWE-377 as a weakness associated with the insecure creation and management of temporary files.\n","title":"CWE-377 - Insecure Temporary File in Java","type":"posts"},{"content":" We will now create a new Vaadin Flow application step by step and create a basic framework for our own projects with this component-based open-source web framework. So, right from the start, the question arises: How can you start with as little effort as possible without avoiding the usual expenses that sometimes come with creating projects?\nHow and where do I start my project? # First we go to the URL start.vaadin.com. Here, we get to an application that allows us to generate a Vaadin project. Here, we choose the option “Hello World Project \u0026quot; out of. As soon as you get to the field “Edit ” on the right side with the mouse, the selection options shown here open. In this article, I will use Settings a) Flow / Java, b) Java, c) Maven, d) Servlet.\nI chose this setting specifically because I don\u0026rsquo;t want to have any additional technological dependencies on other frameworks at this point, making the project as small as possible.\nAfter downloading the packed project, you can unpack it in a folder of your choice and get the complete directory structure necessary for a Maven project. To work with the project, please open the project with the IDE of your choice. I will use the IDE IntelliJ (https://www.jetbrains.com/idea/) from JetBrains in the community version in this article. This is freely available on all relevant platforms (Windows, Linux, and OSX). At this point I am assuming that a JDK and Maven are already installed and configured to work on the development computer.\nAfter the project has been opened, you can do the first test to see whether everything is working. You can do this, for example, on the command line within the project directory with the following Maven command. mvn clean verify If everything runs through to the end without any error messages, you have downloaded all the necessary dependencies from the relevant repositories to your computer using Maven.\nWith the command mvn clean package jetty:run, you can also try out the project straight away. Everything is translated, assembled, and started within the servlet container Jetty. If you now access the URL with a browser, http://localhost:8080/, you will see the Vaadin Flow application.\nNow, you can try out how this application works. by entering something into the input field and then clicking the button labelled “Say hello ” clicks. In my case, I entered my name, “Sven Ruppert”, and activated the button once. The result then looks like this in my browser.\nWhat is included in the project? # But how can you program this yourself? To do this, we first end the ongoing process that we are using mvn have started.\nFirst, we edit the pom.xml a little. Here, we will limit Vaadin\u0026rsquo;s dependency on pure open-source components. This saves time and space during development. For this purpose, the definition “vaadin ” changed in \u0026ldquo;vaadin-core ”\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.vaadin\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;vaadin-core\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; The application logic # The example application consists of three Java classes. We\u0026rsquo;ll now start with the business logic we used in the class GreetingService find. There\u0026rsquo;s not much to say about this, as it\u0026rsquo;s a good old plain Java class. The goal is to transform the input string passed so that the greeting comes out. We don\u0026rsquo;t find any Vaadin elements here yet.\npublic class GreetService { public String greet(String name) { if (name == null || name.isEmpty()) { return \u0026#34;Hello anonymous user\u0026#34;; } else { return \u0026#34;Hello \u0026#34; + name; } } } Then, we are missing two more Java classes, which we can find in the source code folder. On the one hand, it’s the class ”MainView\u0026rdquo;. The entire visual design of this demo app is implemented here.\nHow do I define the GUI? # @Route(\u0026#34;\u0026#34;) public class MainView extends VerticalLayout { public MainView() { TextField textField = new TextField(\u0026#34;Your name\u0026#34;); GreetService greetService = new GreetService(); Button button = new Button(\u0026#34;Say hello\u0026#34;, and -\u0026gt; { String value = textField.getValue(); String greet = greetService.greet(value); Paragraph paragraph = new Paragraph(greet); add(paragraph); }); add(textField, button); } } Let\u0026rsquo;s go through this together. First, we see the class definition “MainView ”, which arises from deriving the class “VerticalLayout ” results. This is a layout component in which the elements are arranged vertically inside. Within the constructor, we now define the various visible elements. In this case, it is an input field and a button. At the very end, these are in the order of text field and button with the method add(..) added to the VerticalLayout. The GreetingService is created within the constructor, and then within the ActionListeners, the button is used. For this example, the procedure shown is exemplary. However, we will examine when you should use other design patterns in a later article. The button here has a clearly defined English label, “Say hello ”, that the user sees. Things get more interesting when defining behaviour. When you click the button, the values ​​are read from the text field using the instance of GreetingServers modified. This new value is then embedded in a graphical component of the Paragraph type and passed to the external layout as a child node.\nHere, you can see quite clearly that you don\u0026rsquo;t have to worry too much about the typical web application challenges during development. You can use everything that the Java language offers in terms of abstractions.\nThe MainView class has an annotation called “@Route(“”) ”. Die Annotation “@Route ” is used in Vaadin to map a view component (e.g. a UI) to a specific URL. It indicates which class is the web application\u0026rsquo;s main view for a particular path. This allows the user interface to be accessed via a specific URL in the browser.\nThe annotation\u0026rsquo;s value specifies the path under which the view can be reached. If no path is specified, the view is accessible at the root path (\u0026quot;/\u0026quot;).\n@Route(\u0026#34;\u0026#34;) public class MainView extends VerticalLayout { // View content } In this example, the **MainView** is under the root path (“http://localhost:8080/ \u0026ldquo;) reachable. You can also specify a specific path to make the view accessible at a specific URL.\n@Route(\u0026#34;all\u0026#34;) public class ToDoView extends VerticalLayout { // View content } The “ToDoView \u0026quot; under the path \u0026ldquo;http://localhost:8080/todo \u0026quot; is reachable in this example.\nThe annotation allows Vaadin to control navigation within the application. Users can navigate to different views by changing the URL in the browser or using programmatic navigation methods. The \u0026ldquo;@Route ” annotation can also be used with a layout. A layout is a high-level component that contains multiple views. Here is an example:\n@Route(value = \u0026#34;todo\u0026#34;, layout = MainLayout.class) public class ToDoView extends VerticalLayout { // View content } In this example, “ToDoView ” within the \u0026ldquo;MainLayout ” is displayed if the URL “http://localhost:8080/todo ” is called. You can also use dynamic parameters in the URL to display specific content in the view.\n@Route(\u0026#34;user/:userID\u0026#34;) public class UserView extends VerticalLayout { // View content } In this example, “UserView ” at a URL like “http://localhost:8080/user/123 ” is called, whereas “123 ” is a dynamic parameter.\nIn summary, the **@Route** annotation defines URL mapping to View classes in Vaadin, which enables navigation and access to various parts of the web application.\nHow can I configure the web app? # Now, we\u0026rsquo;re missing the class AppShell , which is the implementation of the interface AppSehllConfigurator.\n@PWA(name = \u0026#34;Project Base for Vaadin\u0026#34;, shortName = \u0026#34;Project Base\u0026#34;) @Push @Theme(\u0026#34;my-theme\u0026#34;) public class AppShell implements AppShellConfigurator { } The interface ”AppShellConfigurator ” in Vaadin Flow defines the general configuration of the HTML shell of your Vaadin application. This mainly concerns the “\u0026lt; head\u0026gt;” area of ​​the HTML document where meta tags, titles, links to external resources (such as CSS files or favicons), and similar configurations can be set.\nHere, for example, I added “Dark” to the theme definition, which now gives the following look.\n@PWA(name = \u0026#34;Project Base for Vaadin\u0026#34;, shortName = \u0026#34;Project Base\u0026#34;) @Push @Theme(value = \u0026#34;my-theme\u0026#34;, variant = Lumo.DARK) public class AppShell implements AppShellConfigurator { } Let’s go back to the method”configurePage ” in Vaadin Flow. Here, you can configure the general HTML shell of your application. This method customises various aspects of the HTML document, especially the “\u0026lt; head\u0026gt;\u0026quot;-Area. Here are the main adjustments you can make:\nYou can set the HTML page title. The page title is displayed in the browser tab and is essential for user experience and search engine optimisation.\nMeta tags are essential for providing metadata about HTML documents. With \u0026ldquo;configurePage ” you can add different meta tags, such as:\nViewport meta tag : This is especially important for responsive designs as it determines how the page will appear on different devices and screen sizes. Description : A meta tag to describe the content of your page, which is vital for search engine optimisation (SEO). Keywords : Keywords that describe the page\u0026rsquo;s content and can improve SEO. Favicons are tiny icons that appear in browser tabs and bookmarks. You can set different sizes and types of favicons to ensure they look good on all devices and platforms. You can add links to external resources such as CSS stylesheets or JavaScript files. This is useful for including additional libraries or styles in your application.\nFor Progressive Web Apps (PWAs), you can use the “configurePage ” method to configure the manifest and service worker. The manifest defines how the application displays on the home screen of a mobile device while using the service worker for offline functionality and push notifications.\nTo increase the security of your application, you can add various security-related meta tags, such as Content Security Policy (CSP).\nSocial media tags such as Open Graph (for Facebook) or Twitter Cards can be added to determine how your pages appear when shared on social networks. For mobile applications, you can set specific settings that improve the appearance and behaviour of your web application on mobile devices, such as the status bar colour and the app icon on the home screen.\nBy using the method “configurePage ”, you can ensure that all critical meta information and resources are consistently and correctly integrated into every HTML page in your application. This contributes to a better user experience, optimised performance and improved web application security.\nWhich Are there any components? # After discussing the existing classes, we can modify the GUI a bit. We will now try out various components a little. The first question is, what categories of components are there in Vaadin Flow?\nThere are a variety of components in Vaadin Flow that can be grouped into different categories. Here are the main types of components with some examples:\nBasic layout components: # VerticalLayout : Arranges components vertically. HorizontalLayout : Arranges components horizontally. Div : A simple container that works like a \u0026lt;div\u0026gt; element in HTML. FormLayout : Explicitly designed for form layouts to arrange input fields in a grid. Input components: # TextField: Simple text input. TextArea: Multi-line text input. PasswordField: Input field for passwords in which the characters are hidden. ComboBox: A drop-down menu with choices. DatePicker: Selecting a date. CheckBox: A checkbox. RadioButtonGroup: A group of radio buttons. Select: A drop-down selection component. Button components: # Button : A simple button that can be clicked. NativeButton : A button that behaves like a native HTML button. Checkbox : A checkbox. Display and information components: # Label : A simple text label. Html : Component for displaying HTML content. Image : To view images. Notification : To view notifications. ProgressBar : A progress bar. Badge : To display small blocks of information or counters. Navigation components: # MenuBar : A menu bar. Tabs : Tabs for navigation between different views. Accordion : An accordion component for collapsible content. DrawerToggle : A toggle for a side menu. Grid and data displays: # Grid : A robust table for displaying data. TreeGrid : A tree-structured grid for displaying hierarchical data. ListBox : A simple list to display choices. DataView : To display data in a structured form. Dialogues and overlays: # Dialog : A modal dialogue for user interactions. Overlay : To overlay content on the current view. Multimedia components: # Audio : For embedding and playing audio files. Video : For embedding and playing video files. Forms and validation: # FormLayout : Used to arrange form fields in a layout. Binder : For binding forms to data models and validation. Special and advanced components: # Upload : To upload files. RichTextEditor : A WYSIWYG editor for rich text input. SplitLayout : To divide the layout into two areas using a sliding divider. DateTimePicker : To select the date and time. These components cover many use cases and enable developers to create rich and interactive web applications. Vaadin Flow also offers the ability to create and extend custom components to meet specific needs.\nHello World for advanced users # Let\u0026rsquo;s now move on to the practical application of individual components. We are writing a small application that includes multiple input fields, a table to display data, and easy navigation between two views. This extension showcases more of Vaadin\u0026rsquo;s capabilities, including layouts, forms, data bindings, and navigation. However, we have not yet gone into the concepts of industrial applications. We will address this in detail in the next post.\nIf you have different views, the question almost automatically arises as to how you can define the general appearance of the application. Layouts in which the primary division of the application is defined are suitable for this. Vaadin Flow offers you the component AppLayout at this point. This is a simple layout structure that already provides you with elements such as a navigation bar and a workspace. This is precisely what we will use now. For this, we create the class MainLayout and direct them from AppLayout away. In the constructor, we define the desired structure. In our example, we would like to receive two entries for navigation. Firstly, access to the existing MainView and secondly to a new view with the name DataView to record and display a personal data record. The class RouterLink is the required element to be added to a navigation bar.\npublic class MainLayout extends AppLayout { public MainLayout() { createHeader(); } private void createHeader() { RouterLink mainLink = new RouterLink(\u0026#34;Hello World\u0026#34;, MainView.class); RouterLink dataLink = new RouterLink(\u0026#34;Data View\u0026#34;, DataView.class); addToNavbar(new HorizontalLayout(mainLink, dataLink)); } } So the class MainView Now knowing that it is within the layout is in the annotation @Route the class MainLayout specified in the layout attribute.\n@Route(value = \u0026#34;\u0026#34;, layout = MainLayout.class) public class MainView extends VerticalLayout { //COLLAR } This way, a nested layout can be quickly built with Vaadin Flow.\nNow, let\u0026rsquo;s move on to the second view, DataView. Here, we can do the same thing as MainView proceeds.\n@Route(value = \u0026#34;data\u0026#34;, layout = MainLayout.class) public class DataView extends VerticalLayout { private List\u0026lt;Person\u0026gt; people = new ArrayList\u0026lt;\u0026gt;(); private Grid\u0026lt;Person\u0026gt; grid = new Grid\u0026lt;\u0026gt;(Person.class); private ListDataProvider\u0026lt;Person\u0026gt; dataProvider = new ListDataProvider\u0026lt;\u0026gt;(people); public DataView() { was firstNameField = new TextField(\u0026#34;First Name\u0026#34;); was lastNameField = new TextField(\u0026#34;Last Name\u0026#34;); was addButton = new Button(\u0026#34;Add Person\u0026#34;, event -\u0026gt; { people.add( new Person(firstNameField.getValue(), lastNameField.getValue())); dataProvider.refreshAll(); }); grid.setDataProvider(dataProvider); grid.setColumns(\u0026#34;firstName\u0026#34;, \u0026#34;lastName\u0026#34;); add(firstNameField, lastNameField, addButton, grid); } } The DataView is also derived from one VerticalLayout. First, we define the required infrastructure elements. In this case, there is a list for the people, a table to display them, and a ListDataProvider , which is the connection between the graphical representation and the data itself. I won\u0026rsquo;t go into the specific properties here, just this much in advance: the attributes of the class Person are the headings of the columns in the table.\nNow, we define the appearance and dynamic behaviour of the displayed components. This time, there are two text fields for entering the data and a button for adding the data to the table. The ActionListener of the button creates an instance of the class from the values ​​of the two input fields Person. This instance is added to the list and subsequently informed to the DataProvider that there has been a change to the data. So that the DataProvider and the table know about each other, they are made known to each other. This is done by passing the DataProvider to the table.\nFinally, all visible elements are added to the DataView.\nIf you start the application now, you will get the following views:\nWe now have a rudimentary basis for our projects with Vaadin Flow. The following article will look at how we can expand the project.\nI have placed the example on Github using the following URL.\nUntil then, have fun with your experiments.\nHappy Coding\nSven\n","date":"19 June 2024","externalUrl":null,"permalink":"/posts/vaadin-flow-how-to-start/","section":"Posts","summary":" We will now create a new Vaadin Flow application step by step and create a basic framework for our own projects with this component-based open-source web framework. So, right from the start, the question arises: How can you start with as little effort as possible without avoiding the usual expenses that sometimes come with creating projects?\n","title":"Vaadin Flow - How to start","type":"posts"},{"content":" What is Test Coverage? # Test coverage is a metric used in software testing to measure the testing performed on a piece of software. It indicates how thoroughly a software program has been tested by identifying which parts of the code have been executed (covered) during testing and which have not. Here are the key aspects of test coverage:\nWhat is Test Coverage? Purpose: Examples of Types of Test Coverage: Benefits: Limitations: What is the history of Test Coverage? Early Days of Software Testing (1950s-1960s) Introduction of Code Coverage (1970s) Development of Coverage Tools (1980s-1990s) Rise of Agile and Test-Driven Development (2000s) Modern Coverage Tools and Practices (2010s-Present) Key Figures and Influential Works Trends and Future Directions The Basic: Line Coverage Key Concepts of Line Coverage How Line Coverage Works Example Workflow with JaCoCo Interpreting Line Coverage Reports Best Practices Limitations Property Based Testin Coverage Key Concepts of Property-Based Testing Popular Libraries for Property-Based Testing in Java How to Use Property-Based Testing in Java Benefits of Property-Based Testing Challenges and Best Practices Mutation-Based Testing Coverage Key Concepts of Mutation Testing Steps in Mutation Testing Pitest for Mutation Testing in Java Example of Mutation Testing Benefits of Mutation Testing Challenges and Best Practices Conclusion Purpose : # The primary purpose of test coverage is to ensure that the code has been exercised to the maximum extent possible, reducing the chances of undiscovered bugs. It helps identify untested parts of the code, enabling testers to create additional tests to cover those areas.\nExamples of Types of Test Coverage : # Code Coverage : Measures the extent to which the source code is tested. It includes various levels like:\nStatement Coverage : Ensures that each statement in the code has been executed at least once.\nBranch Coverage : Ensures that each branch (true/false) of control structures (like if statements) has been executed.\nFunction Coverage : Ensures that each function in the code has been called and executed.\nCondition Coverage : Ensures that each boolean expression has been evaluated to be both true and false.\nFeature Coverage : Measures how many features or requirements have been tested.\nPath Coverage : Ensures that all possible paths in the code have been executed.\nBenefits: # Improved Quality : Higher test coverage generally leads to higher quality software as more defects are likely to be found and fixed.\nRisk Management : Identifying untested parts of the code helps manage risks by thoroughly testing critical areas.\nMaintenance : Helps maintain the code by testing new changes and not introducing new bugs.\nLimitations: # False Sense of Security : High coverage does not guarantee the software is bug-free. It only indicates that the tests have executed the code.\nFocus on Quantity : Focusing too much on coverage percentages can lead to writing tests that simply increase coverage without effectively testing the functionality.\nComplexity : Achieving 100% coverage can be difficult and time-consuming, especially for large and complex codebases.\nOverall, test coverage is a valuable metric in the software development lifecycle. It provides insights into the effectiveness of the testing process and highlights areas that may need additional attention.\nWhat is the history of Test Coverage? # The history of test coverage can be traced back to the evolution of software engineering and the increasing complexity of software systems, which necessitated more rigorous and systematic testing methodologies. Here are some key milestones and developments in the history of test coverage:\nEarly Days of Software Testing (1950s-1960s) # Initial Testing Practices : In the early days of computing, software testing was often ad hoc, with no formal methods or metrics. Testing was primarily focused on ensuring that the software ran without crashing.\nEmergence of Formal Methods : The need for systematic testing approaches became evident as software systems became more complex. Early researchers and practitioners started formalising testing techniques.\nIntroduction of Code Coverage (1970s) # Code Coverage Concept : Code coverage emerged in the 1970s to measure how thoroughly software tests exercised the code. This period saw the development of various coverage criteria, such as statement coverage, branch coverage, and path coverage.\nMyers\u0026rsquo; Work : Glenford Myers\u0026rsquo; book The Art of Software Testing, first published in 1979, was influential in promoting structured and systematic testing practices, including the use of coverage metrics.\nDevelopment of Coverage Tools (1980s-1990s) # Early Tools : In the 1980s and 1990s, early test coverage tools were developed that automated code coverage measurement. These tools helped developers and testers visualise and quantify the extent of their testing efforts.\nIntegration with Development Environments : As integrated development environments (IDEs) and automated testing frameworks became more common, coverage tools began to be integrated into these environments, making it easier for developers to incorporate coverage measurement into their workflows.\nRise of Agile and Test-Driven Development (2000s) # Agile Methodologies : The adoption of Agile methodologies in the early 2000s emphasised iterative development and continuous testing, leading to a greater focus on test coverage to ensure code quality.\nTest-Driven Development (TDD) : TDD, a practice where tests are written before the code itself, gained popularity. This approach inherently promoted high levels of test coverage, as each piece of code was written to pass specific tests.\nModern Coverage Tools and Practices (2010s-Present) # Advanced Coverage Tools : Modern coverage tools have become more sophisticated, offering features such as real-time coverage analysis, integration with CI/CD pipelines, and detailed reporting. Examples include JaCoCo for Java, Istanbul for JavaScript, and Coverage.py for Python.\nShift-Left Testing : The shift-left testing approach, which advocates for testing early and often in the development lifecycle, has further reinforced the importance of test coverage. Tools and practices have evolved to support this approach, ensuring that coverage metrics are available throughout development.\nCode Quality Platforms : Comprehensive code quality platforms like SonarQube have integrated test coverage metrics with other quality indicators, providing a holistic view of software health.\nKey Figures and Influential Works # Glenford Myers : His seminal work \u0026ldquo;The Art of Software Testing\u0026rdquo; laid the foundation for systematic software testing and introduced many critical concepts related to test coverage.\nTom DeMarco and Tim Lister : Their work on software metrics and quality assurance highlighted the importance of measuring and improving software processes, including testing practices.\nTrends and Future Directions # Increased Automation : As software development continues towards greater automation, test coverage tools are becoming more integrated with automated testing frameworks and CI/CD pipelines.\nAI and Machine Learning : Emerging technologies like AI and machine learning are being applied to testing and coverage analysis, helping to identify untested areas more intelligently and predict potential risks based on coverage data.\nFocus on Test Effectiveness : While achieving high test coverage is important, there is a growing emphasis on test effectiveness. This includes ensuring that tests not only cover code but also validate that the code behaves correctly under various conditions.\nTest coverage has evolved significantly over the decades from a conceptual metric to a critical component of modern software development practices, helping to ensure the quality and reliability of software systems.\nThe Basic: Line Coverage # Line coverage is a code coverage metric that measures whether each line of source code in a program has been executed during testing. In Java, line coverage helps developers understand how thoroughly their tests exercise their code by indicating which lines have been tested and which have not. Here’s an in-depth look at line coverage in Java:\nKey Concepts of Line Coverage # Definition : Statement coverage measures whether each line of code has been executed at least once during testing.\nImportance :\nBug Detection : Helps identify untested parts of the code that might contain bugs. Code Quality : Ensures that all parts of the code are tested, leading to better overall code quality. Maintenance : Aids in maintaining the code by ensuring that changes and new additions are tested. Tools for Measuring Line Coverage in Java :\nJaCoCo (Java Code Coverage) : A widely used open-source library for measuring code coverage in Java projects. Emma : Another popular tool but has mainly been superseded by JaCoCo. Cobertura : An older code coverage tool that is still used in some projects. EclEmma : An Eclipse plugin for JaCoCo, making visualising coverage directly in the IDE easy. How Line Coverage Works # Instrumentation : Tools like JaCoCo instrument the bytecode of a Java program to insert probes at various points. These probes collect data on which lines of code are executed during the test run.\nRunning Tests : Once the code is instrumented, the tests are executed. As the tests run, the probes collect execution data for each line of code.\nReporting : After the tests are complete, the coverage tool generates a report showing which lines of code were executed. This report often includes visual indicators (such as colour coding) to show covered (executed) and uncovered (not executed) lines.\nExample Workflow with JaCoCo # Adding JaCoCo to the Project :\nFor a Maven project, you add the JaCoCo plugin to the pom.xml file:\n\u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.jacoco\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;jacoco-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.8.8\u0026lt;/version\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;prepare-agent\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;id\u0026gt;report\u0026lt;/id\u0026gt; \u0026lt;phase\u0026gt;test\u0026lt;/phase\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;report\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; Running Tests :\nExecute your tests using Maven: mvn clean test\nGenerating Coverage Report :\nAfter running the tests, generate the coverage report: mvn jacoco:report\nViewing the Report :\nThe report is usually generated in the **target/site/jacoco** directory. To view the detailed coverage report, open the index.html file in a web browser.\nInterpreting Line Coverage Reports # Coverage Percentage : Indicates the proportion of lines executed versus the total number of lines.\nHighlighted Lines : Typically, green lines indicate covered lines, while red lines indicate uncovered lines.\nDetailed Metrics : Reports might include additional metrics like the number of covered and missed lines, branches, and methods.\nBest Practices # Aim for High Coverage : While 100% coverage is ideal, strive for the highest coverage practical for your project to ensure maximum reliability.\nFocus on Critical Code : Ensure that the most critical and complex parts of the code are covered extensively.\nRegular Monitoring : Run coverage reports regularly as part of your CI/CD pipeline to catch regressions and ensure new code is adequately tested.\nComplement with Other Testing Techniques : Line coverage is one aspect of testing. For comprehensive coverage, use it alongside other techniques like unit, integration, and manual testing.\nLimitations # False Sense of Security : High-line coverage does not guarantee the absence of bugs. It’s possible to have tests covering all lines but not effectively testing the logic.\nFocus on Quantity over Quality : Simply aiming for high coverage numbers can lead to writing superficial tests that do not profoundly validate the code\u0026rsquo;s behaviour.\nLine coverage is a valuable metric for assessing the thoroughness of your tests in Java projects. Using tools like JaCoCo and following best practices ensures that your code is well-tested and maintains high-quality standards. However, it should be used as part of a broader testing strategy to achieve the best results.\nProperty Based Testin Coverage # Property-based testing (PBT) is a testing methodology where instead of writing individual test cases with specific inputs and expected outputs, you define properties that should hold true for a wide range of inputs. The testing framework then automatically generates test cases to verify these properties. This can uncover edge cases and unexpected behaviours that might not be evident with traditional example-based testing.\nKey Concepts of Property-Based Testing # Properties : These are general assertions about the behaviour of your code. Properties describe the expected behaviour for various inputs rather than specific examples.\nGenerators : These automatically create input data for tests, allowing the exploration of a wide range of possible values.\nShrinkers : When a test fails, shrinkers reduce the size of the failing input to the simplest form that still causes the failure, making it easier to diagnose the problem.\nPopular Libraries for Property-Based Testing in Java # jqwik : A modern property-based testing library for Java that integrates well with JUnit 5.\nQuickTheories : A property-based testing tool inspired by QuickCheck from Haskell, focusing on creating and shrinking data for tests.\nJUnit-QuickCheck : An extension for JUnit that brings property-based testing capabilities inspired by Haskell\u0026rsquo;s QuickCheck to Java.\nHow to Use Property-Based Testing in Java # Adding jqwik to Your Project :\nFor a Maven project, add the following dependency to your pom.xml:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;net.jqwik\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;jqwik\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.5.1\u0026lt;/version\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; Writing a Property-Based Test :\nDefine properties using the **@Property** annotation and use jqwik\u0026rsquo;s built-in generators to create input data.\nimport net.jqwik.api.*; public class ExamplePropertyTest { @Property boolean concatenationLength(@ForAll String a, @ForAll String b) { return (a + b).length() == a.length() + b.length(); } } Running the Tests :\nExecute the tests using your preferred method, such as running through an IDE that supports JUnit 5 or using Maven: mvn test\nInterpreting Results :\nThe framework automatically generates a wide range of inputs to test the property. If a property fails, jqwik attempts to shrink the failing input to a minimal case.\nBenefits of Property-Based Testing # Increased Coverage : Automatically generates a large number of test cases, covering more scenarios and edge cases.\nUncovering Edge Cases : Helps find edge cases that might not be considered during example-based testing.\nLess Manual Work : Reduces the need to manually write extensive individual test cases.\nBetter Specifications : Encourages writing more general and robust specifications of program behaviour.\nChallenges and Best Practices # Defining Good Properties : One of the main challenges is defining general and meaningful properties. These properties should capture the code\u0026rsquo;s essential invariants and behaviours.\nHandling Complex Input Data : It may be necessary to write custom generators and shrinkers for complex input data.\nBalancing Between Unit and Property-Based Tests : Property-based testing is robust but should complement rather than replace traditional unit tests. Unit tests are still useful for specific scenarios and regression testing.\nProperty-based testing in Java provides a powerful tool for verifying that code behaves correctly across a wide range of inputs. Defineting properties and using automated input generation helps uncover bugs and edge cases that might be missed by traditional testing methods. While there are challenges in defining good properties and handling complex inputs, the benefits of increased coverage and robustness make it a valuable addition to any testing strategy.\nMutation-Based Testing Coverage # Mutation testing is a technique used to evaluate the quality of software tests. It involves modifying a program\u0026rsquo;s source code in small ways, called \u0026ldquo;mutants,\u0026rdquo; and then running the test suite to see if the tests detect these changes. A good test suite should be able to detect and fail when these mutants are introduced, indicating that the tests are effective in catching errors.\nKey Concepts of Mutation Testing # Mutants : Variations of the original program created by making small changes, such as altering operators, changing constants, or modifying control flow.\nMutation Operators : Specific rules or patterns used to create mutants. Examples include:\nArithmetic Operator Replacement (AOR) : Replaces arithmetic operators like + with -.\nLogical Operator Replacement (LOR) : This function replaces logical operators like \u0026amp;\u0026amp; with ||.\nRelational Operator Replacement (ROR) : Replaces relational operators like \u0026gt; with \u0026lt;.\nMutation Score : The percentage of mutants that are detected (killed) by the test suite.\nSteps in Mutation Testing # Generate Mutants : Apply mutation operators to the original source code to create a set of mutant programs.\nRun Tests : Execute the test suite on each mutant.\nAnalyse Results : Determine if the tests pass or fail for each mutant. If a test fails, the mutant is considered \u0026ldquo;killed.\u0026rdquo; If all tests pass, the mutant is considered \u0026ldquo;survived.\u0026rdquo;\nPitest for Mutation Testing in Java # A widely used mutation testing tool for Java that integrates with various build tools and CI/CD pipelines.\nAdding PIT to Your Project\nAdd the PIT plugin to your pom.xml:\n\u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.pitest\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;pitest-maven\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.6.9\u0026lt;/version\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;mutationCoverage\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; Running Mutation Tests\nRun the mutation tests using: mvn org.pitest:pitest-maven:mutationCoverage\nInterpreting Results\nPIT generates a detailed HTML report showing the mutation score, which mutants were killed, and which survived. The report helps identify weak spots in your test suite.\nExample of Mutation Testing # Consider a simple Java class and its test:\nCalculator.java :\npublic class Calculator { public int add(int a, int b) { return a + b; } } CalculatorTest.java :\nimport static org.junit.jupiter.api.Assertions.assertEquals; import org.junit.jupiter.api.Test; public class CalculatorTest { @Test public void testAdd() { Calculator calc = new Calculator(); assertEquals(5, calc.add(2, 3)); } } When running mutation testing on this example, a typical mutant might change the addition operator to subtraction:\npublic int add(int a, int b) { return a - b; } If the test suite is adequate, it will fail for this mutant, thus killing it. The test suite might need improvement to cover this scenario if it doesn\u0026rsquo;t.\nBenefits of Mutation Testing # Improves Test Quality : Helps identify weaknesses in the test suite and areas where tests might be missing.\nComprehensive Coverage : Provides a more rigorous evaluation of test effectiveness compared to code coverage metrics alone.\nIdentifies Redundant Tests : Helps detect tests that do not contribute to detecting faults in the code.\nChallenges and Best Practices # Performance Overhead : Mutation testing can be time-consuming, especially for large codebases, as it requires running the test suite multiple times.\nEquivalent Mutants : Some mutants may be logically equivalent to the original code and cannot be killed by any test, which can be challenging to identify and handle.\nSelective Mutation Testing : To reduce the performance overhead, focus on critical parts of the codebase or use a subset of mutation operators.\nMutation testing is a powerful technique for assessing the effectiveness of your test suite by introducing small changes (mutants) to the code and checking if the tests detect these changes. Tools like PIT make it practical to integrate mutation testing into Java projects, providing valuable insights into test quality and helping to improve the robustness of your tests. By complementing traditional testing methods with mutation testing, you can achieve higher confidence in the correctness and reliability of your software.\nConclusion # Each coverage type has its strengths and weaknesses, making them suitable for different aspects of software testing:\nLine Coverage is useful for ensuring that all parts of the code are executed, but should be complemented with other methods to ensure test quality.\nProperty-Based Coverage excels at testing the code\u0026rsquo;s behaviour over a wide range of inputs and can uncover edge cases that traditional testing might miss.\nMutation-Based Coverage provides a rigorous assessment of test suite effectiveness by ensuring that tests can detect intentional faults, though it is resource-intensive.\nFor a comprehensive testing strategy, it is beneficial to combine these approaches, leveraging the strengths of each to ensure both thorough and practical testing.\n","date":"31 May 2024","externalUrl":null,"permalink":"/posts/comparing-code-coverage-techniques-line-property-based-and-mutation-testing/","section":"Posts","summary":"What is Test Coverage? # Test coverage is a metric used in software testing to measure the testing performed on a piece of software. It indicates how thoroughly a software program has been tested by identifying which parts of the code have been executed (covered) during testing and which have not. Here are the key aspects of test coverage:\n","title":"Comparing Code Coverage Techniques: Line, Property-Based, and Mutation Testing","type":"posts"},{"content":"","date":"31 May 2024","externalUrl":null,"permalink":"/tags/jacoco/","section":"Tags","summary":"","title":"Jacoco","type":"tags"},{"content":"","date":"31 May 2024","externalUrl":null,"permalink":"/categories/mutation-testing/","section":"Categories","summary":"","title":"Mutation Testing","type":"categories"},{"content":"","date":"31 May 2024","externalUrl":null,"permalink":"/tags/mutation-testing/","section":"Tags","summary":"","title":"Mutation Testing","type":"tags"},{"content":"","date":"31 May 2024","externalUrl":null,"permalink":"/tags/pitest/","section":"Tags","summary":"","title":"Pitest","type":"tags"},{"content":"","date":"31 May 2024","externalUrl":null,"permalink":"/tags/property-based-testing/","section":"Tags","summary":"","title":"Property Based Testing","type":"tags"},{"content":"","date":"31 May 2024","externalUrl":null,"permalink":"/tags/qwik/","section":"Tags","summary":"","title":"Qwik","type":"tags"},{"content":"","date":"31 May 2024","externalUrl":null,"permalink":"/tags/test-coverage/","section":"Tags","summary":"","title":"Test Coverage","type":"tags"},{"content":"","date":"27 May 2024","externalUrl":null,"permalink":"/tags/infection-method/","section":"Tags","summary":"","title":"Infection Method","type":"tags"},{"content":"","date":"27 May 2024","externalUrl":null,"permalink":"/tags/package-manager/","section":"Tags","summary":"","title":"Package Manager","type":"tags"},{"content":" What is a Package Manager - Bird-Eye View # A package manager is a tool or system in software development designed to simplify the process of installing, updating, configuring, and removing software packages on a computer system. It automates managing dependencies and resolving conflicts between different software components, making it easier for developers to work with various libraries, frameworks, and tools within their projects.\nPackage managers typically provide a centralised repository or repositories where software packages are hosted. Users can then use the package manager to search for, download, and install the desired packages and any necessary dependencies directly from these repositories.\nWhat is a Package Manager - Bird-Eye View What are the pros and cons of Package Managers? Pros: Simplified Installation: Dependency Management: Version Control: Centralised Repository: Consistency: Cons: Limited Control: Security Risks: Versioning Issues: Performance Overhead: Dependency Bloat: What is Maven? Project Object Model (POM): Dependency Management: Convention over Configuration: Build Lifecycle: Plugins: Central Repository: Integration with IDEs: Transparency and Repeatability: What are typical Securit Risks using Maven? Dependency vulnerabilities: Malicious dependencies: Repository compromise: Man-in-the-middle attacks: Build script injection: How does the dependency-resolving mechanism work in Maven? Dependency Declaration: Dependency Tree: Repository Resolution: Dependency Download: Version Conflict Resolution: Transitive Dependency Resolution: Dependency Caching: Dependency Scope: What attacks target Maven\u0026rsquo;s cache structure? Cache Poisoning: Cache Exfiltration: Cache Manipulation: Conclusion: Some popular package managers include:\n1. APT (Advanced Package Tool) : Used primarily in Debian-based Linux distributions such as Ubuntu, APT simplifies installing and managing software packages.\n2. YUM (Yellowdog Updater Modified) and DNF (Dandified YUM) : Package managers commonly used in Red Hat-based Linux distributions like Fedora and CentOS.\n3. Homebrew : A package manager for macOS and Linux, Homebrew simplifies the installation of software packages and libraries.\n4. npm (Node Package Manager) and Yarn are package managers for JavaScript and Node.js that manage dependencies in web development projects.\n5. pip : The package installer for Python, allowing developers to easily install Python libraries and packages from the Python Package Index (PyPI) repository.\n6. Composer: is a dependency manager for PHP, used to manage libraries and dependencies in PHP projects.\nPackage managers greatly simplify software development by automating the process of installing and managing software dependencies. This reduces errors, improves efficiency, and facilitates collaboration among developers.\nWhat are the pros and cons of Package Managers? # Package managers offer numerous benefits to software developers and users alike, but they also have drawbacks. Here are the pros and cons of using a package manager:\nPros: # Simplified Installation : # Package managers automate the process of installing software packages, making it straightforward for users to add new software to their systems.\nDependency Management: # Package managers handle dependencies automatically, ensuring all required libraries and components are installed correctly. This reduces the likelihood of dependency conflicts and makes it easier to manage complex software ecosystems.\nVersion Control: # Package managers keep track of software versions and updates, allowing users to upgrade to newer versions when they become available quickly. This helps ensure that software remains up-to-date and secure.\nCentralised Repository: # Package managers typically provide access to a centralised repository of software packages, making it easy for users to discover new software and libraries.\nConsistency: # Package managers enforce consistency across different environments by ensuring that all installations are performed in a standardised manner. This reduces the likelihood of configuration errors and compatibility issues.\nCons: # Limited Control: # Package managers abstract away many of the details of software installation and configuration, which can sometimes limit users\u0026rsquo; control over the installation process. Advanced users may prefer more manual control over their software installations.\nSecurity Risks: # Because package managers rely on centralised repositories, malicious actors could compromise them and distribute malicious software packages. Users must trust the integrity of the repository and the software packages hosted within it.\nVersioning Issues: # Dependency management can sometimes lead to versioning issues, primarily when multiple software packages depend on conflicting versions of the same library. Resolving these conflicts can be challenging and may require manual intervention.\nPerformance Overhead: # Package managers introduce a performance overhead, as they must download and install software packages and their dependencies. In some cases, this overhead may be negligible, but it can become a concern for large projects or systems with strict performance requirements.\nDependency Bloat: # Dependency management can sometimes lead to \u0026ldquo;dependency bloat,\u0026rdquo; where software projects rely on a large number of external libraries and components. This can increase the size of software installations and introduce additional maintenance overhead.\nWhile package managers offer significant benefits in simplifying software installation and dependency management, users must be aware of the potential drawbacks and trade-offs involved.\nWhat is Maven? # Apache Maven is a powerful build automation and dependency management tool primarily used for Java projects. However, it can also manage projects in other languages like C#, Ruby, and Scala. It provides a consistent and standardised way to build, test, and deploy software projects. Here are some critical details about Maven:\nProject Object Model (POM): # Maven uses a Project Object Model (POM) to describe a project\u0026rsquo;s structure and configuration. The POM is an XML file that contains information about the project\u0026rsquo;s dependencies, build settings, plugins, and other metadata. It serves as the central configuration file for the project and is used by Maven to automate various tasks.\nDependency Management: # One of Maven\u0026rsquo;s key features is its robust dependency management system. Dependencies are specified in the POM file, and Maven automatically downloads the required libraries from remote repositories such as Maven Central. Maven also handles transitive dependencies, automatically resolving and downloading dependencies required by other dependencies.\nConvention over Configuration: # Maven follows the \u0026ldquo;convention over configuration\u0026rdquo; principle, which encourages standard project structures and naming conventions. By adhering to these conventions, Maven can automatically infer settings and reduce the configuration required.\nBuild Lifecycle: # Maven defines a standard build lifecycle consisting of phases: compile, test, package, install, and deploy. Each phase represents a specific stage in the build process, and Maven plugins can be bound to these phases to perform various tasks such as compiling source code, running tests, creating JAR or WAR files, and deploying artefacts.\nPlugins: # Maven is highly extensible through plugins, which provide additional functionality for tasks such as compiling code, generating documentation, and deploying artefacts. Maven plugins can be either built-in or custom-developed, and they can be configured in the POM file to customise the build process.\nCentral Repository: # Maven Central is the default repository for hosting Java libraries and artefacts. It contains a vast collection of open-source and third-party libraries that can be easily referenced in Maven projects. Additionally, organisations and individuals can set up their own repositories to host proprietary or custom libraries.\nIntegration with IDEs: # Maven integrates seamlessly with popular Integrated Development Environments (IDEs) such as Eclipse, IntelliJ IDEA, and NetBeans. IDEs typically provide Maven support through plugins, allowing developers to import Maven projects, manage dependencies, and run Maven goals directly from the IDE.\nTransparency and Repeatability: # Maven uses declarative configuration and standardised project structures to promote transparency and repeatability in the build process. This makes it easier for developers to understand and reproduce builds across different environments.\nOverall, Apache Maven is a versatile and widely used tool in the Java ecosystem. It offers powerful features for automating build processes, managing dependencies, and promoting best practices in software development. Its adoption by numerous open-source projects and enterprises underscores its importance in modern software development workflows.\nWhat are typical Securit Risks using Maven? # While Apache Maven is a widely used and generally secure tool for managing dependencies and building Java projects, there are some potential security risks associated with its usage:\nDependency vulnerabilities: # One of the leading security risks with Maven (as with any dependency management system) is the possibility of including dependencies with known vulnerabilities. If a project relies on a library with a security flaw, it could expose the application to various security risks, including remote code execution, data breaches, and denial-of-service attacks. It\u0026rsquo;s essential to update dependencies to patched versions regularly and use tools like OWASP Dependency-Check to identify and mitigate vulnerabilities.\nMalicious dependencies: # While Maven Central and other reputable repositories have strict guidelines for publishing artefacts, there is still a risk of including malicious or compromised dependencies in a project. Attackers could compromise a legitimate library or create a fake library with malicious code and upload it to a repository. Developers should only use trusted repositories, verify the integrity of dependencies, and review code changes carefully.\nRepository compromise: # Maven relies on remote repositories to download dependencies, and if a repository is compromised, it could serve malicious or tampered artefacts to unsuspecting users. While Maven Central has robust security measures, smaller or custom repositories may have weaker security controls. Organisations should implement secure repository management practices, such as using HTTPS for communication, signing artefacts with GPG, and restricting access to trusted users.\nMan-in-the-middle attacks: # Maven communicates with remote repositories over HTTP or HTTPS, and if an attacker intercepts the communication, they could tamper with the downloaded artefacts or inject malicious code into the project. To mitigate this risk, developers should use HTTPS for all repository communications, verify SSL certificates, and consider using tools like Maven\u0026rsquo;s own repository manager or Nexus Repository Manager, which support repository proxies and caching to reduce the reliance on external repositories.\nBuild script injection: # Maven\u0026rsquo;s build scripts (POM files) are XML files that define project configurations, dependencies, and build settings. If an attacker gains unauthorised access to a project\u0026rsquo;s source code repository or CI/CD pipeline, they could modify the build script to execute arbitrary commands or introduce vulnerabilities into the build process. Organisations should implement proper access controls, code review processes, and security scanning tools to detect and prevent unauthorised changes to build scripts.\nBy understanding these security risks and implementing best practices for secure software development and dependency management, developers can mitigate potential threats and ensure the integrity and security of their Maven-based projects. Regular security audits, vulnerability scanning, and staying informed about security updates and patches are also essential for maintaining a secure development environment.\nHow does the dependency-resolving mechanism work in Maven? # Maven\u0026rsquo;s dependency resolution mechanism is a crucial aspect of its functionality, ensuring that projects have access to the necessary libraries and components while managing conflicts and versioning issues effectively. Here\u0026rsquo;s how the dependency-resolving mechanism works in Maven:\nDependency Declaration: # Developers specify dependencies for their projects in the project\u0026rsquo;s POM (Project Object Model) file. Dependencies are declared within the \u0026lt;dependencies\u0026gt; element, where each dependency includes details such as group ID, artifact ID, and version.\nDependency Tree: # Maven constructs a dependency tree based on the declared dependencies in the POM file. This tree represents the hierarchical structure of dependencies, including direct dependencies (specified in the POM file) and transitive dependencies (dependencies required by other dependencies).\nRepository Resolution: # When a build is initiated, Maven attempts to resolve dependencies by searching for them in configured repositories. By default, Maven Central is the primary repository, but developers can specify additional repositories in the POM file or through Maven settings.\nDependency Download: # If a dependency is not already present in the local Maven repository, Maven downloads it from the remote repository where it\u0026rsquo;s hosted. Maven stores downloaded dependencies in the local repository (~/.m2/repository by default) for future reuse.\nVersion Conflict Resolution: # Maven employs a strategy for resolving version conflicts when multiple dependencies require different versions of the same library. By default, Maven uses the \u0026ldquo;nearest-wins\u0026rdquo; strategy, where the version closest to the project in the dependency tree takes precedence. Developers can explicitly specify versions for dependencies to override the default resolution behaviour. Maven also provides options such as dependency management and exclusions to manage version conflicts better.\nTransitive Dependency Resolution: # Maven automatically resolves transitive dependencies, ensuring all required libraries and components are included in the project\u0026rsquo;s classpath. Maven traverses the dependency tree recursively, downloading and including transitive dependencies as needed.\nDependency Caching: # Maven caches downloaded dependencies in the local repository to improve build performance and reduce network traffic. Subsequent builds reuse cached dependencies whenever possible, avoiding redundant downloads.\nDependency Scope: # Maven supports different dependency scopes (e.g., compile, test, runtime) to control the visibility and usage of dependencies during various phases of the build lifecycle. Scopes help prevent unnecessary dependencies in production builds and improve build efficiency.\nOverall, Maven\u0026rsquo;s dependency resolution mechanism simplifies the management of project dependencies by automating the process of downloading, organizing, and resolving dependencies. This allows developers to focus on writing code rather than manually managing library dependencies.\nWhat attacks target Maven\u0026rsquo;s cache structure? # Attacks targeting the cache structure of Maven, particularly the local repository cache (~/.m2/repository), are relatively rare but can potentially exploit vulnerabilities in the caching mechanism to compromise the integrity or security of Maven-based projects. Here are some potential attacks that could target Maven\u0026rsquo;s cache structure:\nCache Poisoning: # Description : Cache poisoning attacks involve manipulating the contents of the local repository cache to introduce malicious artefacts or modified versions of legitimate artefacts. Once poisoned, subsequent builds may unwittingly use the compromised artefacts, leading to security vulnerabilities or system compromise.\nAttack Vector : Attackers may exploit vulnerabilities in Maven\u0026rsquo;s caching mechanism, such as improper input validation or insecure handling of cached artefacts, to inject malicious artefacts into the cache.\nMitigation : To mitigate cache poisoning attacks, Maven users should regularly verify the integrity of cached artefacts using checksums or digital signatures. Employing secure repository management practices, such as signing artefacts with GPG and enabling repository managers with artefact validation, can also enhance security.\nCache Exfiltration: # Description : Cache exfiltration attacks involve an attacker extracting sensitive information from the local repository cache. This could include credentials, private keys, or other confidential data inadvertently stored within cached artefacts or metadata.\nAttack Vector : Attackers may exploit vulnerabilities in Maven or its plugins to access and extract sensitive information stored in the local repository cache. For example, a compromised plugin could inadvertently leak credentials stored in Maven settings files.\nMitigation : To mitigate cache exfiltration attacks, developers should avoid storing sensitive information in Maven configuration files or cached artefacts. Instead, they should use secure credential management practices, such as environment variables or encrypted credential stores, to prevent the inadvertent exposure of sensitive data.\nCache Manipulation: # Description : Cache manipulation attacks involve modifying the contents of the local repository cache to alter the behaviour of Maven builds or compromise the integrity of projects. To achieve their objectives, attackers may tamper with cached artefacts, metadata, or configuration files.\nAttack Vector : Attackers may exploit vulnerabilities in Maven or its dependencies to manipulate the local repository cache. For example, an attacker could modify cached artefacts to include backdoors or malware.\nMitigation : To mitigate cache manipulation attacks, developers should ensure that the local repository cache is stored in a secure location with restricted access permissions. They should also regularly verify the integrity of cached artifacts and configuration files using checksums or digital signatures. Implementing secure software development practices, such as code reviews and vulnerability scanning, can also help detect and prevent cache manipulation attacks.\nWhile attacks targeting Maven\u0026rsquo;s cache structure are relatively uncommon, developers should remain vigilant and implement security best practices to safeguard against potential vulnerabilities and threats. Regularly updating Maven and its dependencies to the latest versions and maintaining awareness of security advisories and patches is essential for mitigating the risk of cache-related attacks.\nConclusion: # In conclusion, while attacks explicitly targeting Maven\u0026rsquo;s cache structure are relatively rare, they pose potential risks to the integrity and security of Maven-based projects. Cache poisoning, cache exfiltration, and cache manipulation are possible attack vectors that attackers may exploit to compromise Maven\u0026rsquo;s local repository cache (~/.m2/repository).\nTo mitigate these risks, developers and organisations should adopt robust security measures and best practices:\nRegularly Verify Artifact Integrity : Employ mechanisms such as checksums or digital signatures to verify the integrity of cached artefacts and ensure they haven\u0026rsquo;t been tampered with.\nSecure Credential Management : Avoid storing sensitive information, such as credentials or private keys, in Maven configuration files or cached artefacts. Instead, use secure credential management practices, such as environment variables or encrypted stores.\nAccess Control and Permissions : Ensure the local repository cache is stored securely with restricted access permissions to prevent unauthorised access or manipulation.\nUpdate and Patch : Regularly update Maven and its dependencies to the latest versions to mitigate potential vulnerabilities. Stay informed about security advisories and apply patches promptly.\nSecure Repository Management : Implement secure repository management practices, including signing artefacts with GPG, enabling repository managers with artefact validation, and using HTTPS for repository communication.\nBy implementing these security measures and remaining vigilant, developers can reduce the likelihood of cache-related attacks and enhance the overall security posture of Maven-based projects. Additionally, fostering a culture of security awareness and education within development teams can help mitigate risks and respond effectively to emerging threats in the software development lifecycle.\n","date":"27 May 2024","externalUrl":null,"permalink":"/posts/securing-apache-maven-understanding-cache-related-risks/","section":"Posts","summary":"What is a Package Manager - Bird-Eye View # A package manager is a tool or system in software development designed to simplify the process of installing, updating, configuring, and removing software packages on a computer system. It automates managing dependencies and resolving conflicts between different software components, making it easier for developers to work with various libraries, frameworks, and tools within their projects.\n","title":"Securing Apache Maven: Understanding Cache-Related Risks","type":"posts"},{"content":"","date":"22 May 2024","externalUrl":null,"permalink":"/tags/cwe-22/","section":"Tags","summary":"","title":"CWE-22","type":"tags"},{"content":"In today\u0026rsquo;s digital landscape, ensuring the security of your applications is paramount. One critical vulnerability developers must guard against is CWE-22, Path Traversal. This vulnerability can allow attackers to access files and directories outside the intended scope, potentially leading to unauthorised access and data breaches.\nJava\u0026rsquo;s New I/O (NIO) package provides robust tools for file and path manipulation, which can be effectively leveraged to mitigate the risks associated with CWE-22. This blog post will explore best practices for using Java NIO to prevent path traversal vulnerabilities. From proper path normalisation to secure file handling techniques, we\u0026rsquo;ll cover everything you need to know to enhance the security of your Java applications.\nNormalise Paths Validate Path to Ensure It Stays Within a Base Directory Use Secure Directory and File Permissions Setting Permissions for a Directory Setting Permissions for Files Checking File Permissions Practical Security Considerations Handle Symlinks Securely Example: Avoid Following Symlinks Example: Validate the Target of Symlink Practical Security Considerations Validate User Inputs Rigorously Example Implementation Implement Logging and Monitoring Detecting Suspicious Activity Incident Response and Forensics Accountability and Compliance Proactive Threat Detection Improving Security Posture Real-Time Monitoring Use SecureRandom for Temporary Files Use FileSystems for Consistent Path Handling Key Concepts Best Practices Example Implementation Normalise Paths # Normalisation eliminates any redundant elements from a path, such as . (current directory) and .. (parent directory). This process helps ensure that paths are appropriately structured and prevents path traversal attacks.\nCreate the Base Path :\nDefine the base directory against which the user input will be resolved. This should be the directory within which you want to restrict access.\n**Path basePath = Paths.get(\u0026#34;/var/www/uploads\u0026#34;).normalize();** Resolve the User Input :\nCombine the base path with the user-provided input to create a complete path. This step is crucial to ensure that any relative paths the user provides are interpreted in the context of the base path.\nString userInput = request.getParameter(\u0026#34;file\u0026#34;); Path resolvedPath = basePath.resolve(userInput); Normalise the Resolved Path :\nNormalise the resolved path to eliminate any redundant elements. This step ensures that any . or .. in the path are correctly resolved.\n**Path normalizedPath = resolvedPath.normalize();** Validate the Normalized Path :\nEnsure that the normalised path starts with the base path. This check ensures that the path does not traverse outside the intended directory.\nif (!normalizedPath.startsWith(basePath)) { throw new SecurityException(\u0026#34;Invalid file path: path traversal attempt detected.\u0026#34;); } Validate Path to Ensure It Stays Within a Base Directory # Normalise the Base Path :\nEnsure that the base directory is normalised to its simplest form.\nResolve and Normalize the User-Provided Path :\nCombine the base directory with the user-provided path, then normalise the resultant path.\nCheck if the Normalized Path Starts with the Base Path :\nEnsure the final normalised path starts with the base directory to confirm it hasn\u0026rsquo;t traversed outside the intended directory.\nUse Secure Directory and File Permissions # Using secure directory and file permissions is essential to ensuring your files\u0026rsquo; and directories\u0026rsquo; safety and integrity, especially when dealing with user-provided inputs in Java applications.\nJava NIO provides APIs for setting and checking file permissions. The PosixFilePermissions and PosixFileAttributeView classes can do this.\nSetting Permissions for a Directory # To set permissions for a directory, you can use Files.createDirectories and PosixFilePermissions to specify the desired permissions:\nimport java.nio.file.*; import java.nio.file.attribute.PosixFilePermission; import java.nio.file.attribute.PosixFilePermissions; import java.util.Set; import java.io.IOException; public class SecureFileHandler { /** * Creates a directory with secure permissions. * * @param dirPath The path of the directory to create. * @throws IOException if an I/O error occurs. */ public static void createSecureDirectory(Path dirPath) throws IOException { Set\u0026lt;PosixFilePermission\u0026gt; perms = PosixFilePermissions.fromString(\u0026#34;rwxr-x---\u0026#34;); FileAttribute\u0026lt;Set\u0026lt;PosixFilePermission\u0026gt;\u0026gt; attr = PosixFilePermissions.asFileAttribute(perms); Files.createDirectories(dirPath, attr); } /** * Example usage of creating a secure directory. * * @param args Command line arguments. */ public static void main(String[] args) { Path dirPath = Paths.get(\u0026#34;/var/www/uploads\u0026#34;); try { createSecureDirectory(dirPath); } catch (IOException e) { e.printStackTrace(); } } } Setting Permissions for Files # Similarly, you can set permissions for files using the Files.setPosixFilePermissions method:\nimport java.nio.file.*; import java.nio.file.attribute.PosixFilePermission; import java.nio.file.attribute.PosixFilePermissions; import java.io.IOException; import java.util.Set; public class SecureFileHandler { /** * Sets secure permissions for a file. * * @param filePath The path of the file. * @throws IOException if an I/O error occurs. */ public static void setSecureFilePermissions(Path filePath) throws IOException { Set\u0026lt;PosixFilePermission\u0026gt; perms = PosixFilePermissions.fromString(\u0026#34;rw-r-----\u0026#34;); Files.setPosixFilePermissions(filePath, perms); } /** * Example usage of setting secure file permissions. * * @param args Command line arguments. */ public static void main(String[] args) { Path filePath = Paths.get(\u0026#34;/var/www/uploads/example.txt\u0026#34;); try { setSecureFilePermissions(filePath); } catch (IOException e) { e.printStackTrace(); } } } Checking File Permissions # Before performing file operations, checking if the file or directory has the correct permissions is important. Here’s how you can do this:\nimport java.nio.file.*; import java.nio.file.attribute.PosixFilePermissions; import java.nio.file.attribute.PosixFilePermission; import java.io.IOException; import java.util.Set; public class SecureFileHandler { /** * Checks if a file has the required permissions. * * @param filePath The path of the file. * @param requiredPerms The required permissions. * @return True if the file has the required permissions, false otherwise. * @throws IOException if an I/O error occurs. */ public static boolean hasRequiredPermissions(Path filePath, Set\u0026lt;PosixFilePermission\u0026gt; requiredPerms) throws IOException { Set\u0026lt;PosixFilePermission\u0026gt; perms = Files.getPosixFilePermissions(filePath); return perms.containsAll(requiredPerms); } /** * Example usage of checking file permissions. * * @param args Command line arguments. */ public static void main(String[] args) { Path filePath = Paths.get(\u0026#34;/var/www/uploads/example.txt\u0026#34;); Set\u0026lt;PosixFilePermission\u0026gt; requiredPerms = PosixFilePermissions.fromString(\u0026#34;rw-r-----\u0026#34;); try { if (hasRequiredPermissions(filePath, requiredPerms)) { System.out.println(\u0026#34;File has the required permissions.\u0026#34;); } else { System.out.println(\u0026#34;File does not have the required permissions.\u0026#34;); } } catch (IOException e) { e.printStackTrace(); } } } Practical Security Considerations # Restrict Write Access :\nEnsure that write access is limited to necessary users and processes only. This minimises the risk of unauthorised modifications.\nRead-Only Access :\nSet read-only permissions for files that do not need to be modified to prevent unauthorised changes.\nExecute Permissions :\nBe cautious when granting executing permissions. Only grant execute permissions to necessary users and ensure scripts or executables are secure.\nOwner and Group Permissions :\nSet appropriate owner and group permissions. Ensure sensitive files and directories are owned by the correct user and group.\nSymbolic Links :\nAvoid following symbolic links if possible. This can prevent attackers from bypassing security controls using symbolic link attacks.\nUse of umask :\nConfigure the umask value to control the default permission settings for newly created files and directories. This ensures a baseline of security.\nUsing Java NIO’s Path and Files classes and proper permission settings can significantly enhance the security of your file and directory operations. These practices help mitigate the risk of unauthorised access and modifications, protecting your applications from potential vulnerabilities related to CWE-22 (Path Traversal).\nHandle Symlinks Securely # Handling symbolic links (symlinks) securely is crucial to prevent potential security risks such as bypassing access controls and path traversal attacks. Symlinks can be exploited by attackers to gain unauthorised access to files and directories. Here are best practices for securely handling symlinks in Java using the NIO (New I/O) API:\nAvoid Following Symlinks :\nUse the NOFOLLOW_LINKS option when performing file operations to avoid following symlinks. This ensures operations are performed on the symlink rather than the target file or directory.\nValidate the Target of Symlinks :\nIf your application must follow symlinks, validate the symlink\u0026rsquo;s target to ensure it points to an allowed location.\nCheck for Symlinks :\nExplicitly check if a path is a symlink and handle it accordingly.\nExample: Avoid Following Symlinks # Here’s how you can avoid following symlinks in file operations using Java NIO:\nimport java.nio.file.*; import java.nio.file.attribute.BasicFileAttributes; import java.io.IOException; public class SecureSymlinkHandler { /** * Checks if the given path is a symbolic link. * * @param path The path to check. * @return True if the path is a symbolic link, false otherwise. * @throws IOException if an I/O error occurs. */ public static boolean isSymlink(Path path) throws IOException { return Files.isSymbolicLink(path); } /** * Safely deletes a file without following symbolic links. * * @param path The path to the file to delete. * @throws IOException if an I/O error occurs. */ public static void safeDelete(Path path) throws IOException { if (isSymlink(path)) { throw new SecurityException(\u0026#34;Refusing to delete symbolic link: \u0026#34; + path); } Files.delete(path); } /** * Safely reads a file\u0026#39;s attributes without following symbolic links. * * @param path The path to the file. * @return The file\u0026#39;s attributes. * @throws IOException if an I/O error occurs. */ public static BasicFileAttributes safeReadAttributes(Path path) throws IOException { return Files.readAttributes(path, BasicFileAttributes.class, LinkOption.NOFOLLOW_LINKS); } public static void main(String[] args) { Path path = Paths.get(\u0026#34;/var/www/uploads/example.txt\u0026#34;); try { if (isSymlink(path)) { System.out.println(\u0026#34;Path is a symbolic link.\u0026#34;); } else { System.out.println(\u0026#34;Path is not a symbolic link.\u0026#34;); BasicFileAttributes attrs = safeReadAttributes(path); System.out.println(\u0026#34;File size: \u0026#34; + attrs.size()); safeDelete(path); System.out.println(\u0026#34;File deleted safely.\u0026#34;); } } catch (IOException | SecurityException e) { e.printStackTrace(); } } } Example: Validate the Target of Symlink # If your application needs to follow symlinks, validate their targets to ensure they point to an acceptable location\nimport java.nio.file.*; import java.io.IOException; public class SecureSymlinkHandler { /** * Validates that the symlink\u0026#39;s target is within the allowed base directory. * * @param symlink The symbolic link to validate. * @param baseDir The allowed base directory. * @throws IOException if an I/O error occurs or if validation fails. */ public static void validateSymlinkTarget(Path symlink, Path baseDir) throws IOException { if (!Files.isSymbolicLink(symlink)) { throw new IllegalArgumentException(\u0026#34;Path is not a symbolic link: \u0026#34; + symlink); } Path target = Files.readSymbolicLink(symlink).normalize(); Path resolvedTarget = baseDir.resolve(target).normalize(); if (!resolvedTarget.startsWith(baseDir)) { throw new SecurityException(\u0026#34;Invalid symlink target: \u0026#34; + resolvedTarget); } } public static void main(String[] args) { Path symlink = Paths.get(\u0026#34;/var/www/uploads/symlink\u0026#34;); Path baseDir = Paths.get(\u0026#34;/var/www/uploads\u0026#34;).normalize(); try { validateSymlinkTarget(symlink, baseDir); System.out.println(\u0026#34;Symlink target is valid and within the allowed base directory.\u0026#34;); } catch (IOException | SecurityException e) { e.printStackTrace(); } } } Practical Security Considerations # Restrict Symlink Creation :\nOnly allow trusted users to create symlinks. This minimises the risk of symlinks being used for malicious purposes.\nRegularly Audit Symlinks :\nPeriodically audit symlinks within your application to ensure they are not pointing to unauthorised locations.\nUse Symlink Aware Libraries :\nUse libraries that are aware of and handle symlinks securely. This can help mitigate the risk of unintentional symlink following.\nLeast Privilege Principle :\nEnsure that your application runs with the minimum required privileges. This reduces the impact of potential symlink-related vulnerabilities.\nDeploy Defense in Depth :\nUse multiple layers of security controls to protect against symlink attacks. These include filesystem permissions, application-level checks, and regular monitoring.\nValidate User Inputs Rigorously # Validating user inputs rigorously is crucial to ensure the security and integrity of an application. Proper input validation helps prevent various attacks, including path traversal (CWE-22), SQL injection, cross-site scripting (XSS), and more. Here are some best practices and techniques to rigorously validate user inputs in Java applications:\nWhitelist Validation : Only allow inputs that match a predefined set of acceptable values. This is the most secure form of validation.\nBlacklist Validation : Reject inputs that contain known dangerous characters or patterns. This approach is less secure than whitelisting but can be used as an additional measure.\nLength Checks : Ensure that inputs are within the expected length limits. This prevents buffer overflows and denial of service (DoS) attacks.\nData Type Checks : Verify that inputs match the expected data type (e.g., integers, dates).\nEncoding and Escaping : Encode and escape inputs to prevent injection attacks in different contexts (e.g., HTML, SQL).\nCanonicalisation : Convert inputs to a standard format before validation. This helps compare and process inputs securely.\nRestrict Allowed Characters :\nValidate file names and paths against a whitelist of allowed characters. Reject any input containing characters that can alter the path structure (e.g., .., /, \\\\).\nCheck Path Against Base Directory :\nEnsure the resolved and normalised path starts with the intended base directory.\nExample Implementation # Below is an example implementation in Java that combines these principles and techniques to validate user inputs, specifically for file paths:\nimport java.nio.file.*; import java.util.Set; import java.util.HashSet; import java.util.logging.Logger; import java.io.IOException; import java.util.regex.Pattern; public class SecureInputValidator { private static final Logger logger = Logger.getLogger(SecureInputValidator.class.getName()); private static final Set\u0026lt;String\u0026gt; ALLOWED_EXTENSIONS = new HashSet\u0026lt;\u0026gt;(); static { ALLOWED_EXTENSIONS.add(\u0026#34;.txt\u0026#34;); ALLOWED_EXTENSIONS.add(\u0026#34;.jpg\u0026#34;); ALLOWED_EXTENSIONS.add(\u0026#34;.png\u0026#34;); ALLOWED_EXTENSIONS.add(\u0026#34;.pdf\u0026#34;); } /** * Validates the user-provided file name against a whitelist of allowed characters and extensions. * * @param fileName The user-provided file name. * @throws IllegalArgumentException if the file name is invalid. */ public static void validateFileName(String fileName) throws IllegalArgumentException { if (fileName == null || fileName.isEmpty()) { throw new IllegalArgumentException(\u0026#34;File name cannot be null or empty.\u0026#34;); } // Check for invalid characters Pattern pattern = Pattern.compile(\u0026#34;[^a-zA-Z0-9._-]\u0026#34;); if (pattern.matcher(fileName).find()) { throw new IllegalArgumentException(\u0026#34;File name contains invalid characters.\u0026#34;); } // Check for allowed file extensions boolean validExtension = ALLOWED_EXTENSIONS.stream().anyMatch(fileName::endsWith); if (!validExtension) { throw new IllegalArgumentException(\u0026#34;File extension is not allowed.\u0026#34;); } } /** * Validates the user-provided path to ensure it stays within the base directory. * * @param baseDir The base directory. * @param userInput The user-provided input. * @return The validated and normalized path. * @throws SecurityException if a path traversal attempt is detected. * @throws IllegalArgumentException if the file name is invalid. * @throws IOException if an I/O error occurs. */ public static Path getSecureFilePath(String baseDir, String userInput) throws SecurityException, IllegalArgumentException, IOException { validateFileName(userInput); // Normalize the base directory Path basePath = Paths.get(baseDir).normalize(); // Resolve the user input against the base directory and normalize the result Path resolvedPath = basePath.resolve(userInput).normalize(); // Validate that the resolved path starts with the base directory if (!resolvedPath.startsWith(basePath)) { logSuspiciousActivity(userInput); throw new SecurityException(\u0026#34;Invalid file path: path traversal attempt detected.\u0026#34;); } return resolvedPath; } /** * Logs suspicious activity for further analysis. * * @param userInput The suspicious user input. */ private static void logSuspiciousActivity(String userInput) { logger.warning(\u0026#34;Suspicious file access attempt: \u0026#34; + userInput); } /** * Example usage of the secure file path validation. * * @param args Command line arguments. */ public static void main(String[] args) { String baseDir = \u0026#34;/var/www/uploads\u0026#34;; String userInput = \u0026#34;example.txt\u0026#34;; try { Path filePath = getSecureFilePath(baseDir, userInput); System.out.println(\u0026#34;Validated file path: \u0026#34; + filePath); } catch (SecurityException | IllegalArgumentException | IOException e) { e.printStackTrace(); } } } Implement Logging and Monitoring # Logging and monitoring are essential to effectively prevent or deal with CWE-22 (Path Traversal) vulnerabilities. Here\u0026rsquo;s why these practices are critical:\nDetecting Suspicious Activity # Logging and monitoring allow you to detect suspicious activities that might indicate a path traversal attempt. By capturing and analysing logs, you can identify unusual patterns or attempts to access files and directories that should be off-limits.\nExample : Logging all file access attempts, including the requested paths, can help you spot anomalies such as attempts to use ../ sequences to access parent directories.\nprivate static final Logger logger = Logger.getLogger(SecureFileHandler.class.getName()); public static void logFileAccessAttempt(String filePath) { logger.info(\u0026#34;File access attempt: \u0026#34; + filePath); } Incident Response and Forensics # In a security incident, having detailed logs can help in forensic analysis to understand how the attack was carried out. This information is vital for closing security gaps and preventing future attacks.\nExample : Detailed logs can show the sequence of operations leading up to a detected breach, including the exact user inputs and system responses.\ntry { Path filePath = resolveFilePath(userInput); logFileAccessAttempt(filePath.toString()); } catch (SecurityException e) { logger.warning(\u0026#34;Security exception: \u0026#34; + e.getMessage()); throw e; } Accountability and Compliance # Many regulatory frameworks and standards require logging and monitoring as part of their compliance requirements. Implementing these practices ensures that you meet legal and regulatory obligations.\nExample : Compliance with standards such as PCI-DSS or GDPR often requires comprehensive logging of security-related events to ensure accountability.\nProactive Threat Detection # Continuous monitoring allows you to detect and respond to threats proactively before they cause significant damage. Real-time monitoring solutions can alert you to potential path traversal attacks as they occur.\nExample : Implementing real-time alerts for suspicious file access patterns can help take immediate corrective actions.\n// Example of setting up a real-time alert system (pseudo-code) if (detectedPathTraversalAttempt) { alertSecurityTeam(\u0026#34;Potential path traversal attack detected: \u0026#34; + attemptedPath); } Improving Security Posture # Regularly analysing logs and monitoring data helps improve overall security posture by identifying weaknesses and refining security policies.\nExample : Reviewing access logs can highlight frequent user errors or misconfigurations that could be exploited, allowing you to address these proactively.\nReal-Time Monitoring # Use monitoring tools and services to keep track of log data and generate alerts for suspicious activities. Tools like ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, or cloud-based solutions such as AWS CloudWatch and Azure Monitor can be useful.\n// Pseudo-code for setting up real-time monitoring and alerts if (detectPathTraversal(logFileAccessAttempt)) { triggerAlert(\u0026#34;Potential path traversal attack detected: \u0026#34; + logFileAccessAttempt); } Implementing logging and monitoring is not just a best practice but a necessary component of a robust security strategy. It helps in the early detection of threats, compliance with regulatory standards, effective incident response, and overall security posture improvement. Ensuring that all file access attempts and related activities are logged and monitored can better protect your applications from CWE-22 and other security vulnerabilities.\nUse SecureRandom for Temporary Files # Using SecureRandom to create temporary files in Java ensures better security and randomness, reducing the risk of vulnerabilities such as CWE-22. Here are best practices for using SecureRandom for this purpose:\nInstantiate SecureRandom : Create an instance of SecureRandom to generate random values for temporary file names or other purposes requiring secure randomness.\n**SecureRandom secureRandom = new SecureRandom();** Generate Secure Random Values : Use SecureRandom to generate a sequence of bytes that can be converted into a string or used directly in file names.\nbyte[] randomBytes = new byte[16]; secureRandom.nextBytes(randomBytes); String randomString = new BigInteger(1, randomBytes).toString(16); Create Temporary Files with Secure Names : Create temporary files using the random string generated by SecureRandom. This helps prevent predictable file names, reducing the risk of collisions or attacks.\nPath tempDir = Paths.get(System.getProperty(\u0026#34;java.io.tmpdir\u0026#34;)); Path tempFile = tempDir.resolve(\u0026#34;tempFile_\u0026#34; + randomString + \u0026#34;.tmp\u0026#34;); Files.createFile(tempFile); Ensure File Permissions : Set appropriate file permissions to prevent unauthorized users from accessing the temporary files.\n**Files.setPosixFilePermissions(tempFile, PosixFilePermissions.fromString(\u0026#34;rw-------\u0026#34;));** Handle SecureRandom Properly : Ensure that SecureRandom is not reseeded unnecessarily, as it can reduce the randomness quality. Initialise it once and reuse the instance.\nSecureRandom secureRandom = new SecureRandom(); // Use secureRandom throughout the application Use java.nio.file for File Operations : Utilize the NIO package to benefit from its security features and avoid common pitfalls associated with older IO methods.\n**Path tempFile = Files.createTempFile(tempDir, \u0026#34;tempFile_\u0026#34;, \u0026#34;.tmp\u0026#34;);** Use FileSystems for Consistent Path Handling # Using FileSystems in Java for consistent path handling ensures that paths are managed in a compatible way across different platforms and file systems. Here are some best practices and techniques for leveraging FileSystems in Java:\nKey Concepts # FileSystems Class :\nThe java.nio.file.FileSystems class provides factory methods for creating file system instances and obtaining default file systems.\nPath Interface :\nThe java.nio.file.Path interface represents a file or directory path in a platform-independent way.\nStandardise Path Creation :\nUse FileSystems to create paths consistently.\nBest Practices # Obtain the Default FileSystem\nThe default file system corresponds to the platform\u0026rsquo;s file system on which the Java virtual machine runs.\nimport java.nio.file.FileSystem; import java.nio.file.FileSystems; FileSystem defaultFileSystem = FileSystems.getDefault(); Create Paths Using the FileSystem\nCreate Path objects using the FileSystem to ensure paths are created consistently.\nimport java.nio.file.Path; Path path = defaultFileSystem.getPath(\u0026#34;/var/www/uploads\u0026#34;, \u0026#34;example.txt\u0026#34;); Handle Paths Consistently Across Platforms\nUsing FileSystems helps create paths compatible with different operating systems.\nPath windowsPath = defaultFileSystem.getPath(\u0026#34;C:\\\\Users\\\\Public\\\\Documents\u0026#34;, \u0026#34;example.txt\u0026#34;); Path unixPath = defaultFileSystem.getPath(\u0026#34;/home/user/docs\u0026#34;, \u0026#34;example.txt\u0026#34;); Use Paths for Safe File Operations\nUsing Paths from FileSystems ensures that file operations are performed safely and consistently.\nimport java.nio.file.Files; import java.io.IOException; try { if (!Files.exists(path)) { Files.createFile(path); } // Perform file operations } catch (IOException e) { e.printStackTrace(); } Normalise and Resolve Paths\nAlways normalise and resolve paths to avoid relative and symlinks issues.\nPath basePath = defaultFileSystem.getPath(\u0026#34;/var/www/uploads\u0026#34;).normalize(); Path userPath = basePath.resolve(\u0026#34;example.txt\u0026#34;).normalize(); if (!userPath.startsWith(basePath)) { throw new SecurityException(\u0026#34;Path traversal attempt detected\u0026#34;); } Example Implementation # Here is an example demonstrating the use of FileSystems for consistent path handling:\nimport java.nio.file.FileSystem; import java.nio.file.FileSystems; import java.nio.file.Path; import java.nio.file.Files; import java.nio.file.attribute.PosixFilePermissions; import java.io.IOException; public class SecurePathHandler { private static final FileSystem fileSystem = FileSystems.getDefault(); public static Path createSecureTempFile(String prefix, String suffix) throws IOException { Path tempDir = fileSystem.getPath(System.getProperty(\u0026#34;java.io.tmpdir\u0026#34;)); Path tempFile = Files.createTempFile(tempDir, prefix, suffix); Files.setPosixFilePermissions(tempFile, PosixFilePermissions.fromString(\u0026#34;rw-------\u0026#34;)); return tempFile; } public static Path resolveAndValidatePath(String baseDir, String userInput) throws SecurityException, IOException { Path basePath = fileSystem.getPath(baseDir).normalize(); Path resolvedPath = basePath.resolve(userInput).normalize(); if (!resolvedPath.startsWith(basePath)) { throw new SecurityException(\u0026#34;Invalid file path: path traversal attempt detected.\u0026#34;); } return resolvedPath; } public static void main(String[] args) { try { Path tempFile = createSecureTempFile(\u0026#34;tempFile_\u0026#34;, \u0026#34;.tmp\u0026#34;); System.out.println(\u0026#34;Temporary file created: \u0026#34; + tempFile); String baseDir = \u0026#34;/var/www/uploads\u0026#34;; String userInput = \u0026#34;example.txt\u0026#34;; Path filePath = resolveAndValidatePath(baseDir, userInput); System.out.println(\u0026#34;Validated file path: \u0026#34; + filePath); } catch (IOException | SecurityException e) { e.printStackTrace(); } } } Using FileSystems for consistent path handling in Java ensures that your application can handle paths in a platform-independent manner. This helps in creating secure, reliable, and maintainable file-handling code. Following these best practices can prevent common pitfalls such as path traversal vulnerabilities and ensure that file operations are performed safely and consistently.\n","date":"22 May 2024","externalUrl":null,"permalink":"/posts/cwe-22-best-practices-to-use-java-nio/","section":"Posts","summary":"In today’s digital landscape, ensuring the security of your applications is paramount. One critical vulnerability developers must guard against is CWE-22, Path Traversal. This vulnerability can allow attackers to access files and directories outside the intended scope, potentially leading to unauthorised access and data breaches.\n","title":"CWE-22: Best practices to use Java NIO","type":"posts"},{"content":"","date":"21 May 2024","externalUrl":null,"permalink":"/tags/apache-shiro/","section":"Tags","summary":"","title":"Apache Shiro","type":"tags"},{"content":"CWE-22, commonly called \u0026ldquo;Path Traversal,\u0026rdquo; is a vulnerability when an application fails to appropriately limit the paths users can access through a user-provided input. This can allow attackers to access directories and files outside the intended directory, leading to unauthorised access and potential system compromise. This vulnerability is particularly significant in Java applications due to the ubiquitous use of file handling and web resources. This document will delve into the nature of CWE-22, its implications, exploitation methods, and, most importantly, strategies to mitigate such vulnerabilities in Java applications.\nUnderstanding CWE-22 Implications of CWE-22 Exploitation Techniques Mitigation Strategies in Java Case Study: A Vulnerable Java Application Vulnerable Code Secure Code What Java Libraries are Helping against CWE-22 Apache Commons IO OWASP Java Encoder Apache Shiro Spring Security Tika ESAPI (Enterprise Security API) Java NIO (New I/O) Hibernate Validator What CVEs are based on CWE-22 CVE-2020-9484 CVE-2019-0232 CVE-2018-11784 Understanding CWE-22 # CWE-22 is categorised under \u0026ldquo;Improper Limitation of a Pathname to a Restricted Directory (\u0026lsquo;Path Traversal\u0026rsquo;).\u0026rdquo; This occurs when an application constructs a path using user input without proper validation or sanitisation, allowing the user to specify paths that traverse outside the intended directory. Sequences such as ../ can be used to move up the directory structure.\nFor example, consider the following Java code snippet that reads a file based on user input:\nString fileName = request.getParameter(\u0026#34;file\u0026#34;); File file = new File(\u0026#34;/var/www/uploads/\u0026#34; + fileName); FileInputStream fis = new FileInputStream(file); If the fileName parameter is not properly validated, an attacker can manipulate it to access files outside the /var/www/uploads/ directory, such as /etc/passwd, by using a path traversal sequence (../../etc/passwd).\nImplications of CWE-22 # The implications of CWE-22 can be severe, ranging from unauthorised access to sensitive files to full system compromise. Some of the potential impacts include:\nExposure of Sensitive Information : Attackers can access sensitive files such as configuration files, passwords, and personal data.\nData Integrity Compromise : Attackers can modify files, potentially leading to data corruption or alteration of critical application files.\nDenial of Service (DoS) : Attackers can disrupt normal operations by accessing and modifying system files.\nExecution of Arbitrary Code : In extreme cases, attackers might execute arbitrary code if they gain access to executable files or scripts.\nExploitation Techniques # To exploit a path traversal vulnerability, an attacker typically uses special characters and patterns in the input to navigate the file system. Here are some standard techniques:\nDot-Dot-Slash (../) : The most common method where the attacker uses ../ to move up the directory structure.\nEncoded Characters : Attackers may use URL encoding (%2e%2e%2f) or other encoding schemes to bypass simple input filters.\nNull Byte Injection : Sometimes, null bytes (%00) are used to terminate strings early, effectively ignoring any appended extensions or path components.\nMitigation Strategies in Java # Mitigating CWE-22 in Java involves a combination of secure coding practices, input validation, and proper use of APIs. Here are some detailed strategies:\nCanonicalisation and Normalization :\nEnsure that the file paths are normalised before use. Java provides methods to normalise paths, which can help mitigate traversal attacks.\nimport java.nio.file.Paths; import java.nio.file.Path; public File getFile(String fileName) throws IOException { Path basePath = Paths.get(\u0026#34;/var/www/uploads/\u0026#34;); Path filePath = basePath.resolve(fileName).normalize(); if (!filePath.startsWith(basePath)) { throw new SecurityException(\u0026#34;Attempted path traversal attack detected\u0026#34;); } return filePath.toFile(); } Input Validation and Sanitization :\nPerform strict validation on user inputs. Reject any input that contains potentially malicious patterns such as ../, URL-encoded sequences, or other traversal patterns.\npublic String sanitizeFileName(String fileName) { if (fileName == null || fileName.contains(\u0026#34;..\u0026#34;) || fileName.contains(\u0026#34;/\u0026#34;) || fileName.contains(\u0026#34;\\\\\u0026#34;)) { throw new IllegalArgumentException(\u0026#34;Invalid file name\u0026#34;); } return fileName; } Whitelist Approach :\nUse a whitelist approach to validate filenames against a set of allowed values or patterns. This can be more effective than blacklisting known destructive patterns.\npublic boolean isValidFileName(String fileName) { return fileName.matches(\u0026#34;[a-zA-Z0-9._-]+\u0026#34;); } File Access Controls :\nRestrict file permissions on the server to limit access only to necessary files and directories. This reduces the impact of potential exploits.\nLogging and Monitoring :\nImplement comprehensive logging and monitoring to detect and respond to suspicious activities. Logs should capture sufficient details to trace potential exploitation attempts.\nimport java.util.logging.Logger; public class FileAccessLogger { private static final Logger LOGGER = Logger.getLogger(FileAccessLogger.class.getName()); public void logAccessAttempt(String fileName) { LOGGER.warning(\u0026#34;Attempted access to file: \u0026#34; + fileName); } } Case Study: A Vulnerable Java Application # Let’s consider a hypothetical case study to illustrate the application of these mitigation strategies. Suppose we have a simple Java web application that allows users to download files from the server.\nVulnerable Code # @WebServlet(\u0026#34;/download\u0026#34;) public class FileDownloadServlet extends HttpServlet { protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { String fileName = request.getParameter(\u0026#34;file\u0026#34;); File file = new File(\u0026#34;/var/www/uploads/\u0026#34; + fileName); if (file.exists()) { FileInputStream fis = new FileInputStream(file); response.setContentType(\u0026#34;application/octet-stream\u0026#34;); response.setHeader(\u0026#34;Content-Disposition\u0026#34;, \u0026#34;attachment; filename=\\\u0026#34;\u0026#34; + file.getName() + \u0026#34;\\\u0026#34;\u0026#34;); byte[] buffer = new byte[4096]; int bytesRead; while ((bytesRead = fis.read(buffer)) != -1) { response.getOutputStream().write(buffer, 0, bytesRead); } fis.close(); } else { response.sendError(HttpServletResponse.SC_NOT_FOUND); } } } This servlet reads the file parameter from the request and constructs a File object. Without proper validation, an attacker can exploit this to download arbitrary files from the server.\nSecure Code # @WebServlet(\u0026#34;/download\u0026#34;) public class FileDownloadServlet extends HttpServlet { protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { String fileName = sanitizeFileName(request.getParameter(\u0026#34;file\u0026#34;)); Path basePath = Paths.get(\u0026#34;/var/www/uploads/\u0026#34;); Path filePath = basePath.resolve(fileName).normalize(); if (!filePath.startsWith(basePath)) { response.sendError(HttpServletResponse.SC_FORBIDDEN, \u0026#34;Invalid file path\u0026#34;); return; } File file = filePath.toFile(); if (file.exists()) { FileInputStream fis = new FileInputStream(file); response.setContentType(\u0026#34;application/octet-stream\u0026#34;); response.setHeader(\u0026#34;Content-Disposition\u0026#34;, \u0026#34;attachment; filename=\\\u0026#34;\u0026#34; + file.getName() + \u0026#34;\\\u0026#34;\u0026#34;); byte[] buffer = new byte[4096]; int bytesRead; while ((bytesRead = fis.read(buffer)) != -1) { response.getOutputStream().write(buffer, 0, bytesRead); } fis.close(); } else { response.sendError(HttpServletResponse.SC_NOT_FOUND); } } private String sanitizeFileName(String fileName) { if (fileName == null || fileName.contains(\u0026#34;..\u0026#34;) || fileName.contains(\u0026#34;/\u0026#34;) || fileName.contains(\u0026#34;\\\\\u0026#34;)) { throw new IllegalArgumentException(\u0026#34;Invalid file name\u0026#34;); } return fileName; } } In the secure version, the sanitizeFileName method ensures the file name is free from traversal sequences, and the path is normalised and checked to prevent directory traversal.\nCWE-22, or path traversal, is a critical vulnerability that can have severe consequences if not adequately mitigated. In Java applications, it is essential to employ a combination of secure coding practices, input validation, canonicalisation, and access controls to safeguard against this threat. By understanding the nature of CWE-22 and implementing robust security measures, developers can significantly reduce the risk of unauthorised file access and protect their applications from potential exploitation.\nWhat Java Libraries are Helping against CWE-22 # Several Java libraries and tools can help developers prevent CWE-22 (Path Traversal) vulnerabilities by providing robust input validation, path normalisation, and security mechanisms. Here are some of the most notable ones:\nApache Commons IO # Apache Commons IO provides a variety of utilities for working with the file system, which can help prevent path traversal vulnerabilities.\nFilenameUtils : This class contains methods for normalising and validating file paths.\nimport org.apache.commons.io.FilenameUtils; String basePath = \u0026#34;/var/www/uploads/\u0026#34;; String fileName = FilenameUtils.normalize(request.getParameter(\u0026#34;file\u0026#34;)); if (!FilenameUtils.directoryContains(basePath, basePath + fileName)) { throw new SecurityException(\u0026#34;Attempted path traversal attack detected\u0026#34;); } OWASP Java Encoder # The OWASP Java Encoder library encodes untrusted input to help protect against injection attacks, including those involving path traversal.\nEncoder : Provides methods to encode user input for various contexts, including filenames, safely.\nimport org.owasp.encoder.Encode; String safeFileName = Encode.forJava(request.getParameter(\u0026#34;file\u0026#34;)); Apache Shiro # Apache Shiro is a powerful security framework that provides robust access control mechanisms for enforcing file access policies.\nPath Permissions : Shiro allows you to define fine-grained access control policies related to file access.\n// Define file access permissions in Shiro configuration [urls] /var/www/uploads/** = authc, perms[\u0026#34;file:read\u0026#34;] Spring Security # Spring Security is a comprehensive security framework that integrates seamlessly with Spring applications, providing mechanisms for authentication and authorisation.\nAccess Control : Spring Security can be configured to enforce strict access controls on file resources.\n@PreAuthorize(\u0026#34;hasPermission(#filePath, \u0026#39;read\u0026#39;)\u0026#34;) public void readFile(Path filePath) { // read file logic } Tika # Apache Tika is a library for detecting and extracting metadata and content from various file types. It includes utilities for working with file paths safely.\nTika IOUtils : Utility methods for safe file operations.\nimport org.apache.tika.io.IOUtils; String safeFileName = IOUtils.toString(request.getParameter(\u0026#34;file\u0026#34;), StandardCharsets.UTF_8); ESAPI (Enterprise Security API) # The OWASP ESAPI library provides a comprehensive set of security controls, including file-handling utilities that can help prevent path traversal attacks.\nValidator : Use ESAPI\u0026rsquo;s Validator to validate filenames.\nimport org.owasp.esapi.ESAPI; import org.owasp.esapi.Validator; Validator validator = ESAPI.validator(); String safeFileName = validator.getValidInput(\u0026#34;file name\u0026#34;, request.getParameter(\u0026#34;file\u0026#34;), \u0026#34;Filename\u0026#34;, 255, false); Java NIO (New I/O) # Java NIO provides modern APIs for file operations, including path manipulation and validation.\nPath and Files : Use java.nio.file.Path and java.nio.file.Files for secure file operations.\nimport java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.Files; Path basePath = Paths.get(\u0026#34;/var/www/uploads/\u0026#34;); Path filePath = basePath.resolve(request.getParameter(\u0026#34;file\u0026#34;)).normalize(); if (!filePath.startsWith(basePath)) { throw new SecurityException(\u0026#34;Attempted path traversal attack detected\u0026#34;); } if (Files.exists(filePath)) { // read file logic } Hibernate Validator # Hibernate Validator, the reference implementation of the Bean Validation API, can be used to enforce validation constraints on user inputs.\nCustom Constraints : Define custom validation constraints for filenames.\nimport javax.validation.constraints.Pattern; public class FileRequest { @Pattern(regexp = \u0026#34;[a-zA-Z0-9._-]+\u0026#34;) private String fileName; // getters and setters } The Java libraries and tools provide robust mechanisms to prevent CWE-22 (Path Traversal) vulnerabilities. Using these libraries, developers can ensure that user inputs are appropriately validated, paths are normalised, and access controls are strictly enforced. This multi-layered approach significantly reduces the risk of unauthorised file access and enhances the overall security of Java applications.\nWhat CVEs are based on CWE-22 # Several Common Vulnerabilities and Exposures (CVEs) related to Java applications have been reported that are based on CWE-22, the Improper Limitation of a Pathname to a Restricted Directory (\u0026lsquo;Path Traversal\u0026rsquo;). These CVEs highlight path traversal vulnerabilities\u0026rsquo; real-world implications and impact in various Java-based systems and libraries. Here are some notable examples:\nCVE-2020-9484 # Description : Apache Tomcat HTTP/2 Request Smuggling and Path Traversal.\nAffected Versions : Apache Tomcat 9.0.0.M1 to 9.0.35, 8.5.0 to 8.5.55, and 7.0.0 to 7.0.104.\nDetails : This vulnerability allowed an attacker to upload files to arbitrary locations via a specially crafted request. The issue was related to the improper handling of user input in file upload paths, leading to a path traversal vulnerability.\nMitigation : Upgrade to the latest versions of Apache Tomcat that include the patch for this vulnerability.\nCVE-2019-0232 # Description : Apache Tomcat Remote Code Execution via CGI Servlet.\nAffected Versions : Apache Tomcat 9.0.0.M1 to 9.0.17, 8.5.0 to 8.5.39, and 7.0.0 to 7.0.93.\nDetails : This CVE was related to a path traversal vulnerability that allowed attackers to achieve remote code execution by manipulating the CGI Servlet configuration.\nMitigation : Disable the CGI Servlet if it is unnecessary or upgrade to a version that fixes the vulnerability.\nCVE-2018-11784 # Description : Apache JMeter Path Traversal Vulnerability.\nAffected Versions : Apache JMeter 3.0 to 4.0.\nDetails : This vulnerability allowed attackers to access files outside the intended directories via a path traversal attack, leveraging improper file path validation in the application.\nMitigation : Upgrade to Apache JMeter 5.0 or later versions where the issue has been resolved.\nThese CVEs highlight the importance of addressing CWE-22 vulnerabilities in Java applications. Regularly updating libraries, frameworks, and application code is crucial to mitigating such vulnerabilities. Developers should adopt best practices for input validation, path normalisation, and robust security configurations to protect against path traversal attacks.\n","date":"21 May 2024","externalUrl":null,"permalink":"/posts/cwe-22-improper-limitation-of-a-pathname-to-a-restricted-directory/","section":"Posts","summary":"CWE-22, commonly called “Path Traversal,” is a vulnerability when an application fails to appropriately limit the paths users can access through a user-provided input. This can allow attackers to access directories and files outside the intended directory, leading to unauthorised access and potential system compromise. This vulnerability is particularly significant in Java applications due to the ubiquitous use of file handling and web resources. This document will delve into the nature of CWE-22, its implications, exploitation methods, and, most importantly, strategies to mitigate such vulnerabilities in Java applications.\n","title":"CWE-22: Improper Limitation of a Pathname to a Restricted Directory","type":"posts"},{"content":"","date":"21 May 2024","externalUrl":null,"permalink":"/tags/esapi/","section":"Tags","summary":"","title":"ESAPI","type":"tags"},{"content":"","date":"21 May 2024","externalUrl":null,"permalink":"/tags/hibernate-validator/","section":"Tags","summary":"","title":"Hibernate Validator","type":"tags"},{"content":"","date":"21 May 2024","externalUrl":null,"permalink":"/tags/owasp/","section":"Tags","summary":"","title":"OWASP","type":"tags"},{"content":"","date":"21 May 2024","externalUrl":null,"permalink":"/tags/tika/","section":"Tags","summary":"","title":"Tika","type":"tags"},{"content":"","date":"17 May 2024","externalUrl":null,"permalink":"/tags/cwe-416/","section":"Tags","summary":"","title":"CWE-416","type":"tags"},{"content":" CWE-416: Use After Free # Use After Free (UAF) is a vulnerability that occurs when a program continues to use a pointer after it has been freed. This can lead to undefined behaviour, including crashes, data corruption, and security vulnerabilities. The problem arises because the memory referenced by the pointer may be reallocated for other purposes, potentially allowing attackers to exploit the situation.\nCWE-416: Use After Free Causes: Consequences: CWE-416 in Java Scenario 1: Mismanagement of External Resources Automatic Resource Management with Try-With-Resources: Explicit Nullification: Scenario 2: Inappropriate Use of Weak References Strong References for Critical Data: Check References Before Use: What CVE´s are based on CWE-416 in Java CVE-2022-45146: CVE-2023-38703: What Design Patterns are used to avoid CWE-416 in Java? RAII (Resource Acquisition Is Initialization) Pattern Factory Pattern Singleton Pattern Observer Pattern Decorator Pattern RAII (Resource Acquisition Is Initialization) Pattern in Java without using try-with-resources Example: RAII Implementation without try-with-resources Important Considerations: Cleaner API Example: More details about java.lang.ref.Cleaner, please How to Use java.lang.ref.Cleaner: Example: Benefits: Causes: # Incorrect memory management : Forgetting to nullify pointers after freeing memory.\nDangling pointers : Retaining references to freed memory and accessing it later.\nDouble freeing : Freeing memory more than once, causing subsequent operations on the exact memory location.\nConsequences: # Security vulnerabilities : Attackers can exploit UAF to execute arbitrary code, escalate privileges, or cause denial of service.\nData corruption : UAF can lead to unexpected changes in data, causing program instability or incorrect behaviour.\nProgram crashes : Accessing freed memory can cause segmentation faults or other runtime errors.\nCWE-416 in Java # With its automatic garbage collection, Java inherently protects against many types of memory management issues, including Use After Free (UAF) vulnerabilities. However, UAF-like issues can still occur in Java if resources other than memory (like file handles or network connections) are mismanaged. Below are some scenarios in Java that resemble UAF and strategies to mitigate them.\nScenario 1: Mismanagement of External Resources # In Java, even though memory is managed by the garbage collector, other resources like file handles or network sockets can be mismanaged, leading to use-after-close problems.\nExample :\nimport java.io.*; public class UseAfterFreeExample { public static void main(String[] args) { FileInputStream fis = null; try { fis = new FileInputStream(\u0026#34;example.txt\u0026#34;); // Use the file input stream... } catch (IOException e) { e.printStackTrace(); } finally { if (fis != null) { try { fis.close(); } catch (IOException e) { e.printStackTrace(); } } } // Attempt to use the file input stream after it has been closed try { int data = fis.read(); // This will throw an IOException } catch (IOException e) { System.out.println(\u0026#34;Attempted to read from a closed stream.\u0026#34;); } } } In this example, after closing the FileInputStream, attempting to read from it results in an IOException.\nAutomatic Resource Management with Try-With-Resources: # Use the try-with-resources statement introduced in Java 7, which ensures each resource is closed at the end of the statement.\nimport java.io.*; public class UseAfterFreeFixed { public static void main(String[] args) { try (FileInputStream fis = new FileInputStream(\u0026#34;example.txt\u0026#34;)) { // Use the file input stream... } catch (IOException e) { e.printStackTrace(); } // fis is automatically closed here // Attempting to use fis here will cause a compilation error } } Explicit Nullification: # Set references to null after closing them to avoid accidental reuse.\nimport java.io.*; public class UseAfterFreeWithNull { public static void main(String[] args) { FileInputStream fis = null; try { fis = new FileInputStream(\u0026#34;example.txt\u0026#34;); // Use the file input stream... } catch (IOException e) { e.printStackTrace(); } finally { if (fis != null) { try { fis.close(); } catch (IOException e) { e.printStackTrace(); } finally { fis = null; // Prevent reuse } } } // fis is now null, so the following attempt to use it will fail at compile-time if (fis != null) { try { int data = fis.read(); // This will not compile since fis is null } catch (IOException e) { System.out.println(\u0026#34;Attempted to read from a closed stream.\u0026#34;); } } } } Scenario 2: Inappropriate Use of Weak References # While not a direct UAF issue, misusing weak references can lead to problems where objects are accessed after being reclaimed by the garbage collector.\nimport java.lang.ref.WeakReference; public class WeakReferenceExample { public static void main(String[] args) { Object obj = new Object(); WeakReference\u0026lt;Object\u0026gt; weakRef = new WeakReference\u0026lt;\u0026gt;(obj); obj = null; // Eligible for garbage collection System.gc(); // Suggest garbage collection // Attempt to use the object after it may have been collected if (weakRef.get() != null) { System.out.println(\u0026#34;Object is still alive.\u0026#34;); } else { System.out.println(\u0026#34;Object has been garbage collected.\u0026#34;); } } } In this example, after nullifying the strong reference and suggesting garbage collection, accessing the weak reference may result in null, indicating the object has been collected.\nStrong References for Critical Data: # Avoid using weak references for critical objects that are still needed.\nCheck References Before Use: # Always check if a weak reference has been cleared before using it.\nif (weakRef.get() != null) { // Safe to use the object Object obj = weakRef.get(); // Use obj } else { // Handle the case where the object has been collected System.out.println(\u0026#34;Object has been garbage collected.\u0026#34;); } While Java’s automatic memory management reduces the risk of traditional UAF vulnerabilities, developers must still be cautious with resource management and weak references to avoid similar issues. Modern Java features like try-with-resources and being mindful of object references can help maintain robust and error-free code.\nWhat CVE´s are based on CWE-416 in Java # Here are some notable CVEs related to CWE-416: Use After Free vulnerabilities in Java:\nCVE-2022-45146: # This vulnerability is found in the FIPS Java API of Bouncy Castle BC-FJA before version 1.0.2.4. Changes in the JVM garbage collector in Java 13 and later versions can trigger this issue, causing temporary keys used by the module to be zeroed out while still in use. This results in errors or potential information loss. FIPS compliance users are unaffected as it applies to Java 7, 8, and 11.\nCVE-2023-38703: # This CVE affects the PJSIP multimedia communication library, written in multiple languages, including Java. The issue arises when the synchronisation between higher-level SRTP media and lower-level transport, such as UDP, is not maintained, leading to a use-after-free vulnerability. This can result in unexpected application termination or memory corruption.\nThese examples highlight the importance of proper memory and resource management in Java applications, even though the language manages memory through garbage collection. For more information on these vulnerabilities, visit the National Vulnerability Database (NVD) or specific advisories on platforms like GitHub and mvnrepository.\nWhat Design Patterns are used to avoid CWE-416 in Java? # Several design patterns and best practices can be employed to ensure robust and safe memory and resource management in Java to avoid CWE-416 (Use After Free). While Java\u0026rsquo;s garbage collector reduces the risk of traditional UAF vulnerabilities, the following patterns and practices help avoid related issues with resources like file handles, sockets, or other external resources.\nRAII (Resource Acquisition Is Initialization) Pattern # RAII is a design pattern that ties resource management to object lifetime. Although RAII is more commonly associated with C++, Java can leverage a similar approach using try-with-resources.\ntry (FileInputStream fis = new FileInputStream(\u0026#34;example.txt\u0026#34;)) { // Use the file input stream... } catch (IOException e) { e.printStackTrace(); } // fis is automatically closed here The try-with-resources statement ensures that each resource is closed at the end of the statement, reducing the risk of resource leaks and use-after-close issues.\nFactory Pattern # The Factory Pattern encapsulates an object\u0026rsquo;s creation logic, allowing for central management of resource creation and ensuring that resources are properly initialized and managed.\npublic class ConnectionFactory { public static Connection createConnection() { // Implement resource creation and management return new Connection(); } } // Usage Connection connection = ConnectionFactory.createConnection(); This pattern centralises resource management, making it easier to enforce consistent resource-handling practices.\nSingleton Pattern # The Singleton Pattern ensures that a class has only one instance and provides a global point of access to it. This is particularly useful for managing shared resources like database connections or thread pools.\npublic class DatabaseConnection { private static DatabaseConnection instance; private DatabaseConnection() { // Initialize the connection } public static synchronized DatabaseConnection getInstance() { if (instance == null) { instance = new DatabaseConnection(); } return instance; } } By controlling a single instance, the Singleton Pattern helps manage the lifecycle and closure of shared resources properly.\nObserver Pattern # The Observer Pattern can notify interested parties about resource lifecycle events, such as the availability or closing of a resource. This ensures that all application parts know the resource state and can act accordingly.\npublic interface ResourceListener { void onResourceClosed(); } public class Resource { private List\u0026lt;ResourceListener\u0026gt; listeners = new ArrayList\u0026lt;\u0026gt;(); public void addListener(ResourceListener listener) { listeners.add(listener); } public void close() { // Close the resource notifyListeners(); } private void notifyListeners() { for (ResourceListener listener : listeners) { listener.onResourceClosed(); } } } Observers are notified of resource state changes, allowing them to update their behaviour accordingly.\nDecorator Pattern # The Decorator Pattern allows for dynamic functionality extensions, including resource management features like logging or additional cleanup actions when resources are closed.\npublic class ResourceDecorator implements AutoCloseable { private final AutoCloseable resource; public ResourceDecorator(AutoCloseable resource) { this.resource = resource; } @Override public void close() throws Exception { // Additional cleanup actions resource.close(); } } // Usage try (ResourceDecorator decoratedResource = new ResourceDecorator(new FileInputStream(\u0026#34;example.txt\u0026#34;))) { // Use the resource } This pattern enhances the resource management behaviour without modifying the original resource classes.\nDevelopers can use these patterns to ensure better resource management and mitigate the risk of use-after-free vulnerabilities in Java applications.\nRAII (Resource Acquisition Is Initialization) Pattern in Java without using try-with-resources # RAII (Resource Acquisition Is Initialization) is a design pattern that binds the lifecycle of resources to the lifetime of objects, ensuring resources are adequately released when the object goes out of scope. While Java does not directly support RAII due to its garbage collection system and lack of deterministic destruction, you can still achieve similar behaviour without using try-with-resources by employing custom management classes and finalisers.\nHere’s how you can implement RAII in Java using a custom resource management class and finalisers:\nExample: RAII Implementation without try-with-resources # Create a Resource Management Class :\nThis class will manage the resource and ensure it is adequately released when the object is no longer needed.\npublic class ManagedResource { private boolean isReleased = false; public ManagedResource() { // Acquire the resource System.out.println(\u0026#34;Resource acquired\u0026#34;); } public void useResource() { if (isReleased) { throw new IllegalStateException(\u0026#34;Resource has been released\u0026#34;); } // Use the resource System.out.println(\u0026#34;Resource is being used\u0026#34;); } public void release() { if (!isReleased) { // Release the resource System.out.println(\u0026#34;Resource released\u0026#34;); isReleased = true; } } @Override protected void finalize() throws Throwable { try { release(); } finally { super.finalize(); } } } Use the Resource Management Class :\nEnsure the resource is managed correctly by explicitly calling the release method or relying on the finaliser.\npublic class RAIIExample { public static void main(String[] args) { ManagedResource resource = new ManagedResource(); try { resource.useResource(); // Perform operations with the resource } finally { resource.release(); } // Alternatively, without explicit release, the finalizer will handle it ManagedResource resource2 = new ManagedResource(); resource2.useResource(); // No explicit release call } } Explanation :\nResource Management Class : The ManagedResource class acquires the resource during initialisation and provides a useResource method to use it. The release method releases the resource.\nFinalizer : The finalize method ensures the resource is released when the ManagedResource object is garbage collected if release was not called explicitly.\nUsage : In the RAIIExample class, the resource is explicitly released in the finally block. For the second resource (resource2), if not explicitly released, the finalizer will handle it.\nImportant Considerations: # Finalizers : Using finalizers can be unpredictable, as the timing of garbage collection is not guaranteed. Due to performance implications and unpredictability, it is generally recommended to avoid relying on finalizers for critical resource management.\nExplicit Resource Management : Whenever possible, explicitly manage resources using well-defined methods or blocks (like try-with-resources) to ensure timely and predictable resource release.\nJava Cleaner API : For better control over resource management, consider using the Java Cleaner API introduced in Java 9. This API provides a more flexible and reliable alternative to finalizers.\nCleaner API Example: # import java.lang.ref.Cleaner; public class ManagedResource { private static final Cleaner cleaner = Cleaner.create(); private final Cleaner.Cleanable cleanable; private boolean isReleased = false; public ManagedResource() { this.cleanable = cleaner.register(this, this::release); System.out.println(\u0026#34;Resource acquired\u0026#34;); } public void useResource() { if (isReleased) { throw new IllegalStateException(\u0026#34;Resource has been released\u0026#34;); } System.out.println(\u0026#34;Resource is being used\u0026#34;); } public void release() { if (!isReleased) { System.out.println(\u0026#34;Resource released\u0026#34;); isReleased = true; } } @Override protected void finalize() throws Throwable { try { release(); } finally { super.finalize(); } } } The Cleaner API provides a more controlled and predictable way to manage resources without relying on the finalisation mechanism. It ensures that resources are cleaned up promptly when the object becomes unreachable.\nUsing these techniques, you can effectively manage resources in Java, ensuring they are properly released and avoiding issues like CWE-416.\nMore details about java.lang.ref.Cleaner, please # The java.lang.ref.Cleaner class, introduced in Java 9, is a modern replacement for the deprecated java.lang.ref.Finalizer mechanism. It provides a more flexible and efficient way to clean up resources when an object becomes unreachable.\nHow to Use java.lang.ref.Cleaner: # Create a Cleaner Instance :\nCleaner cleaner = Cleaner.create(); Define a Cleanable Resource :\nA cleanable resource must implement a Runnable interface to define the cleanup action.\nclass State implements Runnable { @Override public void run() { // Cleanup action System.out.println(\u0026#34;Resource cleaned up\u0026#34;); } } Register the Resource with Cleaner :\nRegister the resource and its cleanup action with the Cleaner.\nclass ManagedResource { private final Cleaner.Cleanable cleanable; public ManagedResource(Cleaner cleaner) { State state = new State(); this.cleanable = cleaner.register(this, state); System.out.println(\u0026#34;Resource acquired\u0026#34;); } } Using the Managed Resource :\npublic class CleanerExample { public static void main(String[] args) { Cleaner cleaner = Cleaner.create(); ManagedResource resource = new ManagedResource(cleaner); // Use the resource... // No explicit cleanup is required, the Cleaner will handle it } } Example: # Here’s a complete example demonstrating the usage of java.lang.ref.Cleaner:\nimport java.lang.ref.Cleaner; public class CleanerExample { static class State implements Runnable { @Override public void run() { // Cleanup action System.out.println(\u0026#34;Resource cleaned up\u0026#34;); } } static class ManagedResource { private final Cleaner.Cleanable cleanable; public ManagedResource(Cleaner cleaner) { State state = new State(); this.cleanable = cleaner.register(this, state); System.out.println(\u0026#34;Resource acquired\u0026#34;); } } public static void main(String[] args) { Cleaner cleaner = Cleaner.create(); ManagedResource resource = new ManagedResource(cleaner); // Use the resource... // No explicit cleanup required, the Cleaner will handle it } } Benefits: # Predictability : Cleaner ensures that cleanup actions are executed promptly, unlike finalizers which have unpredictable execution.\nPerformance : Using Cleaner avoids the performance issues associated with finalizers, such as increased GC pressure and potential for memory leaks.\nSimplicity : The API is straightforward, making it easier to implement and maintain.\nBy using java.lang.ref.Cleaner, Java developers can achieve efficient and predictable resource management, minimising risks associated with manual resource cleanup.\n","date":"17 May 2024","externalUrl":null,"permalink":"/posts/cwe-416-use-after-free-vulnerabilities-in-java/","section":"Posts","summary":"CWE-416: Use After Free # Use After Free (UAF) is a vulnerability that occurs when a program continues to use a pointer after it has been freed. This can lead to undefined behaviour, including crashes, data corruption, and security vulnerabilities. The problem arises because the memory referenced by the pointer may be reallocated for other purposes, potentially allowing attackers to exploit the situation.\n","title":"CWE-416: Use After Free Vulnerabilities in Java","type":"posts"},{"content":" Transverse Mercator Projection : # The Gauss-Krüger system uses the transverse Mercator projection, which means the cylindrical projection is rotated 90 degrees. This allows for better accuracy over long north-south extents.\nTransverse Mercator Projection: Ellipsoid: Zones: Coordinates: Carl Friedrich Gauss (1777-1855) Johann Heinrich Louis Krüger (1857-1923) Development of the Gauss-Krüger Coordinate System Technical Features Modern Developments and Usage Step-by-Step Conversion Process Example Conversion Online Tools: Using Java: Ellipsoid : # The system is based on an ellipsoid model of the Earth, which is more accurate than a spherical model. Different regions might use slightly different ellipsoids, but a common one used in Europe is the Bessel 1841 ellipsoid.\nZones : # The Gauss-Krüger system divides the area into longitudinal zones that are 3° wide. Each zone has its own central meridian. This helps to reduce distortions within each zone. Zone numbering usually starts at a prime meridian (often 9° E or 15° E) and increases by 3° for each zone.\nCoordinates : # Coordinates are expressed in meters. The system uses false easting and false northing to ensure that all coordinates within a zone are positive.\nEasting (X) : Measured in meters from the zone\u0026rsquo;s central meridian.\nNorthing (Y) : Measured in meters from the equator.\nAccuracy and Usage : # The Gauss-Krüger system is beneficial for large-scale (detailed) maps due to its high accuracy over short distances. It is commonly used in civil engineering, cadastral mapping, and various geospatial applications within Germany and neighbouring countries.\nConversion to WGS84 :\nConverting Gauss-Krüger coordinates to the more globally used WGS84 system (used in GPS) requires specific transformation parameters and sometimes complex algorithms due to the differences in ellipsoid and projection methods.\nSoftware and Tools :\nVarious GIS software (like ArcGIS, QGIS) and online tools can perform these transformations. They typically require inputting the zone number, the easting, and the northing to transform the coordinates accurately.\nHistory of the Gauss-Krüger Coordinate System # The Gauss-Krüger coordinate system has its roots in the early development of geodesy and map projection techniques in the 19th century. Here is a detailed history of the Gauss-Krüger coordinate system:\nCarl Friedrich Gauss (1777-1855) # Contribution to Mathematics and Geodesy :\nCarl Friedrich Gauss, a German mathematician and physicist, made significant contributions to many fields, including geodesy, the science of measuring and understanding the Earth\u0026rsquo;s geometric shape. Gauss developed the mathematical foundations for the projection that bears his name, the transverse Mercator projection, which is essential for creating accurate maps of regions with large north-south extents.\nTransverse Mercator Projection :\nGauss\u0026rsquo;s work on the transverse Mercator projection provided a method to project the Earth\u0026rsquo;s surface onto a plane with minimal distortion over relatively small areas. This projection uses a cylinder rotated 90 degrees, touching the Earth along a chosen meridian.\nJohann Heinrich Louis Krüger (1857-1923) # Refinement and Application :\nJohann Heinrich Louis Krüger, a German geodesist, refined Gauss\u0026rsquo;s projection method and applied it to practical mapping needs. Krüger\u0026rsquo;s refinements improved the projection\u0026rsquo;s mathematical accuracy, making it more suitable for detailed surveying and mapping work.\nDevelopment of the Gauss-Krüger Coordinate System # Germany and Central Europe :\nThe Gauss-Krüger coordinate system was adopted primarily in Germany and other Central European countries for detailed topographic and cadastral mapping. The system divides the region into longitudinal zones, each 3° wide, with a central meridian. This minimises distortion and ensures high accuracy over small areas.\nEllipsoid Models :\nThe system uses specific ellipsoid models, such as the Bessel 1841 ellipsoid, which closely approximates the shape of the Earth in these regions.\nTechnical Features # False Easting and Northing :\nThe system applies a false easting and northing to ensure that all coordinates within a zone are positive. Typically, the central meridian is assigned a false easting of 500,000 meters, and the equator is assigned a false northing.\nZone-Based System :\nEach zone has its coordinate system, reducing the complexity of calculations and distortions. The zones are numbered, usually starting from a prime meridian (e.g., 9°E) and increasing by 3° for each subsequent zone.\nModern Developments and Usage # Integration with Global Systems :\nWith the advent of global positioning systems (GPS) and the universal adoption of the WGS84 ellipsoid, many regions have transitioned to the UTM (Universal Transverse Mercator) system for broader compatibility. However, the Gauss-Krüger system is still used for specific applications requiring high precision and historical continuity in countries like Germany.\nSoftware and Digital Mapping :\nModern GIS (Geographic Information Systems) software supports the Gauss-Krüger projection, allowing for easy conversion between coordinate systems and integration with global datasets.\nThe Gauss-Krüger coordinate system is a significant development in the history of cartography and geodesy. It combines the foundational work of Carl Friedrich Gauss and the practical refinements of Johann Heinrich Louis Krüger. Its precise and detailed approach to mapping has made it a valuable tool in Europe, particularly for topographic and cadastral mapping. While global systems like UTM are now more widely used, the Gauss-Krüger system remains integral to geospatial history and practice.\nHow do you convert Coordinates into UTM Coordinates? # Converting Gauss-Krüger coordinates to UTM (Universal Transverse Mercator) coordinates involves a few steps. Both systems are based on the transverse Mercator projection but use different parameters and zone definitions. Here’s a step-by-step guide on how to perform this conversion:\nStep-by-Step Conversion Process # Identify the Gauss-Krüger Zone :\nDetermine the Gauss-Krüger zone of your coordinates. Gauss-Krüger zones are typically 3° wide.\nCentral Meridian of Gauss-Krüger Zone :\nEach Gauss-Krüger zone has a central meridian, which is usually a multiple of 3° (e.g., 9°E, 12°E, 15°E, etc.).\nTranslate to Geodetic Coordinates (Latitude and Longitude) :\nConvert the Gauss-Krüger coordinates (easting and northing) to geodetic coordinates (latitude and longitude). This requires:\nThe ellipsoid parameters (e.g., Bessel 1841 for Germany). The false easting (usually 500,000 meters). Applying the inverse transverse Mercator projection. Determine the UTM Zone :\nDetermine the appropriate UTM zone for the longitude obtained from the geodetic coordinates. UTM zones are 6° wide.\nConvert to UTM Coordinates :\nConvert the geodetic coordinates to UTM coordinates using:\nThe WGS84 ellipsoid parameters (commonly used for UTM). The UTM zone central meridian. Applying the transverse Mercator projection to obtain the UTM easting and northing. Example Conversion # Let\u0026rsquo;s walk through an example to make it more transparent.\nGiven :\nGauss-Krüger coordinates: Easting = 3550000 meters, Northing = 5800000 meters. Gauss-Krüger zone central meridian: 12°E (assuming it\u0026rsquo;s in zone 4)\nTranslate Gauss-Krüger to Latitude and Longitude :\nUse the inverse Gauss-Krüger projection to convert (3550000, 5800000) to latitude and longitude. Due to the complexity of the inverse projection, this step typically requires software or detailed formulas.\nExample Result :\nLet\u0026rsquo;s assume the resulting geodetic coordinates are:\nLatitude: 52.0°N Longitude: 13.0°E Determine UTM Zone :\nLongitude 13.0°E falls in UTM zone 33U (UTM zones range from 1 to 60, each 6° wide).\nConvert to UTM Coordinates :**\nUse the transverse Mercator projection with WGS84 parameters and the central meridian of zone 33 (15°E) to convert latitude 52.0°N and longitude 13.0°E to UTM coordinates.\nOnline Tools: # Websites like https://epsg.io/ or other geospatial transformation tools can also perform these conversions.\nUsing Java: # To convert Gauss-Krüger coordinates to UTM coordinates in Java, you can use libraries such as Proj4j, which is a Java port of the “PROJ.4” library (https://de.wikipedia.org/wiki/PROJ.4) used for performing cartographic transformations. Here’s how you can perform the conversion step-by-step:\nAdd Proj4j Library to Your Project :\nIf you are using Maven, add the following dependency to your pom.xml:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.locationtech.proj4j\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;proj4j\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.1.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; Define the Coordinate Reference Systems :\nYou will need to define the Gauss-Krüger and UTM coordinate reference systems.\nPerform the Conversion :\nConvert Gauss-Krüger coordinates to geodetic coordinates (latitude and longitude). Convert the geodetic coordinates to UTM coordinates.\nHere is an example Java program that demonstrates this process:\nimport org.locationtech.proj4j.CRSFactory; import org.locationtech.proj4j.CoordinateReferenceSystem; import org.locationtech.proj4j.ProjCoordinate; import org.locationtech.proj4j.Projection; import org.locationtech.proj4j.ProjectionFactory; import org.locationtech.proj4j.ProjectionTransform; public class GaussKrugerToUTMConverter { public static void main(String[] args) { // Create a CRSFactory instance CRSFactory factory = new CRSFactory(); // Define the Gauss-Krüger CRS (EPSG:31468 for zone 4) CoordinateReferenceSystem gaussKrugerCRS = factory.createFromName(\u0026#34;EPSG:31468\u0026#34;); // Define the UTM CRS (EPSG:32633 for UTM zone 33N) CoordinateReferenceSystem utmCRS = factory.createFromName(\u0026#34;EPSG:32633\u0026#34;); // Define the source coordinates in Gauss-Krüger (example values) double gkEasting = 3550000; double gkNorthing = 5800000; // Create a ProjCoordinate object for the input coordinates ProjCoordinate gkCoord = new ProjCoordinate(gkEasting, gkNorthing); // Create a ProjCoordinate object for the intermediate geodetic coordinates ProjCoordinate geoCoord = new ProjCoordinate(); // Create a ProjectionTransform object to convert from Gauss-Krüger to geodetic ProjectionTransform gkToGeoTransform = new ProjectionTransform(gaussKrugerCRS.getProjection(), ProjectionFactory.getGeographic()); // Transform Gauss-Krüger to geodetic coordinates (longitude, latitude) gkToGeoTransform.transform(gkCoord, geoCoord); // Print the geodetic coordinates (longitude, latitude) System.out.println(\u0026#34;Geodetic coordinates: Longitude = \u0026#34; + geoCoord.x + \u0026#34;, Latitude = \u0026#34; + geoCoord.y); // Create a ProjCoordinate object for the final UTM coordinates ProjCoordinate utmCoord = new ProjCoordinate(); // Create a ProjectionTransform object to convert from geodetic to UTM ProjectionTransform geoToUtmTransform = new ProjectionTransform(ProjectionFactory.getGeographic(), utmCRS.getProjection()); // Transform geodetic coordinates to UTM coordinates geoToUtmTransform.transform(geoCoord, utmCoord); // Print the UTM coordinates (easting, northing) System.out.println(\u0026#34;UTM coordinates: Easting = \u0026#34; + utmCoord.x + \u0026#34;, Northing = \u0026#34; + utmCoord.y); } } Dependencies :\nThe proj4j library is added as a dependency to handle coordinate transformations.\nCoordinate Reference Systems (CRS) :\nThe Gauss-Krüger CRS is defined using EPSG:31468 for zone 4. The UTM CRS is defined using EPSG:32633 for UTM zone 33N.\nTransformation Process :\nA ProjCoordinate object is created for the input Gauss-Krüger coordinates. The Gauss-Krüger coordinates are transformed into geodetic coordinates (longitude and latitude). The geodetic coordinates are then transformed to UTM coordinates.\nNotes :\nEnsure the epsg codes are correct for your specific regions and projections. The actual geodetic transformation can be complex due to ellipsoid differences, so precision might require specific parameters and fine-tuning.\n","date":"16 May 2024","externalUrl":null,"permalink":"/posts/basics-of-the-gauss-kruger-coordinate-system/","section":"Posts","summary":"Transverse Mercator Projection : # The Gauss-Krüger system uses the transverse Mercator projection, which means the cylindrical projection is rotated 90 degrees. This allows for better accuracy over long north-south extents.\n","title":"Basics of the Gauss-Krüger Coordinate System","type":"posts"},{"content":"","date":"16 May 2024","externalUrl":null,"permalink":"/tags/bessel-1841/","section":"Tags","summary":"","title":"Bessel 1841","type":"tags"},{"content":"","date":"16 May 2024","externalUrl":null,"permalink":"/tags/ellipsoid/","section":"Tags","summary":"","title":"Ellipsoid","type":"tags"},{"content":"","date":"16 May 2024","externalUrl":null,"permalink":"/tags/gauss-kr%C3%BCger-coordinates/","section":"Tags","summary":"","title":"Gauss-Krüger Coordinates","type":"tags"},{"content":"","date":"16 May 2024","externalUrl":null,"permalink":"/categories/navigation/","section":"Categories","summary":"","title":"Navigation","type":"categories"},{"content":"","date":"16 May 2024","externalUrl":null,"permalink":"/tags/navigation/","section":"Tags","summary":"","title":"Navigation","type":"tags"},{"content":"","date":"15 May 2024","externalUrl":null,"permalink":"/tags/cwe-787/","section":"Tags","summary":"","title":"CWE-787","type":"tags"},{"content":"The term \u0026ldquo;CWE-787: Out-of-bounds Write \u0026quot; likely refers to a specific security vulnerability or error in software systems. Let\u0026rsquo;s break down what it means:\nOut-of-bounds Write : This is a type of vulnerability where a program writes data outside the boundaries of pre-allocated fixed-length buffers. This can corrupt data, crash the program, or lead to the execution of malicious code.\nCWE-787 : This is a specific identifier for the vulnerability. Identifiers like these are often used in vulnerability databases or bug-tracking systems. They help uniquely identify and reference a particular issue.\nExplanation of Out-of-bounds Write Impact of Out-of-bounds Write Examples and Mitigation Example in C: Mitigation: Example of Safe Code: And how is it done in Java? Understanding Out-of-bounds Write in Java Example of Out-of-bounds Write Mitigation Strategies Corrected Example Common Causes of Out-of-bounds Write in Java Advanced Example Mitigated Version CWE 787 - Based on OffHeap techniques in Java 8 Off-heap Memory Management in Java Example of Out-of-bounds Write with Off-heap Memory Using sun.misc.Unsafe Mitigating Out-of-bounds Write with Off-heap Memory Corrected Example with sun.misc.Unsafe Using java.nio.ByteBuffer Handling Larger Data Structures Example with IntBuffer Explanation OffHeap usage in Java 21 Critical Features for Off-Heap Usage in Java 21 Foreign Function \u0026amp; Memory API Example Using Foreign Function \u0026amp; Memory API Enhanced ByteBuffer Operations Memory Segments and Access VarHandles Example Using Memory Segments Differences in OffHeap usage between Java 17 and Java 21 Foreign Function \u0026amp; Memory API Enhanced ByteBuffer Operations Memory Segments and VarHandles Summary of Differences Explanation of Out-of-bounds Write # An out-of-bounds write occurs when a program writes data past the end, or before the beginning, of the buffer it was allocated. This usually happens due to programming errors such as:\nIncorrectly calculating buffer sizes. Failing to check the bounds before writing data. Using unsafe functions that do not perform bounds checking. Impact of Out-of-bounds Write # The consequences of an out-of-bounds write can be severe, including:\nData Corruption : Overwriting adjacent memory locations can corrupt other data.\nProgram Crashes : Writing to illegal memory addresses can cause the program to crash.\nSecurity Vulnerabilities : Attackers can exploit these vulnerabilities to execute arbitrary code, leading to potential breaches, data leaks, or system compromise.\nExamples and Mitigation # Example in C: # #include \u0026lt;string.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; int main() { char buffer[10]; strcpy(buffer, \u0026#34;This string is too long for the buffer\u0026#34;); printf(\u0026#34;%s\\n\u0026#34;, buffer); return 0; } In this example, the string copied into buffer exceeds its allocated size, causing an out-of-bounds write.\nMitigation: # Bounds Checking : Ensure that any write operations do not exceed the allocated buffer size.\nUse Safe Functions : Utilise safer library functions like strncpy instead of strcpy.\nStatic Analysis : Use static analysis tools to detect out-of-bounds writes during development.\nRuntime Protection : Employ runtime protections such as address space layout randomisation (ASLR) and stack canaries.\nExample of Safe Code: # #include \u0026lt;string.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; int main() { char buffer[10]; strncpy(buffer, \u0026#34;This string is too long for the buffer\u0026#34;, sizeof(buffer) - 1); buffer[sizeof(buffer) - 1] = \u0026#39;\\0\u0026#39;; // Ensure null termination printf(\u0026#34;%s\\n\u0026#34;, buffer); return 0; } In this corrected version, strncpy ensures that no more than the allocated size of buffer is written, and null-terminating the string prevents buffer overflow.\nUnderstanding and preventing out-of-bounds writes is crucial in developing secure and stable software. Proper coding practices, safe functions, and security measures can mitigate these vulnerabilities effectively.\nAnd how is it done in Java? # Out-of-bounds write vulnerabilities are less common in Java than in languages like C or C++ due to built-in safety features, such as automatic bounds checking for arrays. However, certain situations can still lead to such vulnerabilities, often related to misusing arrays or improper handling of indices.\nUnderstanding Out-of-bounds Write in Java # In Java, out-of-bounds write errors typically occur when you try to access an array or a list with an index that is either negative or greater than or equal to the size of the array or list. This usually results in an ArrayIndexOutOfBoundsException.\nExample of Out-of-bounds Write # public class OutOfBoundsWriteExample { public static void main(String[] args) { int[] numbers = new int[5]; for (int i = 0; i \u0026lt;= numbers.length; i++) { // Off-by-one error numbers[i] = i * 2; // This will throw an ArrayIndexOutOfBoundsException } } } In this example, the loop runs from 0 to numbers.length, one past the last valid index. This causes an ArrayIndexOutOfBoundsException.\nMitigation Strategies # Proper Bounds Checking : Always ensure that any access to arrays or lists is within valid bounds.\nEnhanced for Loop : Use enhanced for loops when possible to avoid index errors.\nLibraries and Functions : Use Java\u0026rsquo;s standard libraries and methods that handle bounds checking automatically.\nCorrected Example # public class OutOfBoundsWriteExample { public static void main(String[] args) { int[] numbers = new int[5]; for (int i = 0; i \u0026lt; numbers.length; i++) { // Correct bounds numbers[i] = i * 2; } for (int number : numbers) { System.out.println(number); } } } In this corrected version, the loop runs from 0 to numbers.length—1, which prevents the ArrayIndexOutOfBoundsException.\nCommon Causes of Out-of-bounds Write in Java # Off-by-one Errors : These occur when loops iterate one time too many or too few.\nIncorrect Index Calculation : When the calculation of an index is erroneous due to logic errors.\nManual Array Management : Errors often happen when manipulating arrays directly rather than using collections like ArrayList.\nAdvanced Example # Consider a scenario where an attempt to manage a buffer directly leads to an out-of-bounds write:\npublic class BufferOverflowExample { public static void main(String[] args) { int[] buffer = new int[10]; try { writeToBuffer(buffer, 10, 42); // Trying to write outside the buffer } catch (ArrayIndexOutOfBoundsException e) { System.out.println(\u0026#34;Caught exception: \u0026#34; + e); } } public static void writeToBuffer(int[] buffer, int index, int value) { buffer[index] = value; // Potential out-of-bounds write } } In this example, writeToBuffer is called with an index that is out of bounds, leading to an ArrayIndexOutOfBoundsException.\nMitigated Version # public class BufferOverflowExample { public static void main(String[] args) { int[] buffer = new int[10]; try { writeToBuffer(buffer, 10, 42); // Trying to write outside the buffer } catch (IllegalArgumentException e) { System.out.println(\u0026#34;Caught exception: \u0026#34; + e); } } public static void writeToBuffer(int[] buffer, int index, int value) { if (index \u0026lt; 0 || index \u0026gt;= buffer.length) { throw new IllegalArgumentException(\u0026#34;Index out of bounds: \u0026#34; + index); } buffer[index] = value; } } Here, writeToBuffer checks the index before writing to the buffer, preventing an out-of-bounds write.\nWhile Java provides many safeguards against out-of-bounds writes, developers must still practice diligent bounds checking and utilise safe coding practices to prevent these errors. Handling array indices and leveraging Java\u0026rsquo;s robust standard libraries can help maintain secure and stable code.\nCWE 787 - Based on OffHeap techniques in Java 8 # \u0026ldquo;off-heap\u0026rdquo; in Java refers to memory allocated outside the Java heap, typically managed by native code. Working with off-heap memory can provide performance benefits, especially for large data sets or when interacting with native libraries. Still, it also introduces risks such as out-of-bounds writes.\nOff-heap Memory Management in Java # Java provides several ways to work with off-heap memory, including the sun.misc.Unsafe class and java.nio.ByteBuffer. The sun.misc.Unsafe class allows low-level, unsafe operations, and should be used with caution. The java.nio.ByteBuffer class is safer but requires careful bounds checking.\nExample of Out-of-bounds Write with Off-heap Memory # Using sun.misc.Unsafe # import sun.misc.Unsafe; import java.lang.reflect.Field; public class OffHeapExample { private static final Unsafe unsafe; private static final long BYTE_ARRAY_BASE_OFFSET; static { try { Field field = Unsafe.class.getDeclaredField(\u0026#34;theUnsafe\u0026#34;); field.setAccessible(true); unsafe = (Unsafe) field.get(null); BYTE_ARRAY_BASE_OFFSET = unsafe.arrayBaseOffset(byte[].class); } catch (Exception e) { throw new RuntimeException(e); } } public static void main(String[] args) { long size = 10; long memoryAddress = unsafe.allocateMemory(size); try { for (int i = 0; i \u0026lt;= size; i++) { // Intentional off-by-one error unsafe.putByte(memoryAddress + i, (byte) i); } } finally { unsafe.freeMemory(memoryAddress); } } } In this example, the loop iterates one time too many, causing an out-of-bounds write when i equals size.\nMitigating Out-of-bounds Write with Off-heap Memory # Bounds Checking : Always ensure that memory accesses are within the allocated bounds.\nAbstraction : Use higher-level abstractions that handle bounds checking automatically, such as java.nio.ByteBuffer.\nCorrected Example with sun.misc.Unsafe # import sun.misc.Unsafe; import java.lang.reflect.Field; public class OffHeapExample { private static final Unsafe unsafe; private static final long BYTE_ARRAY_BASE_OFFSET; static { try { Field field = Unsafe.class.getDeclaredField(\u0026#34;theUnsafe\u0026#34;); field.setAccessible(true); unsafe = (Unsafe) field.get(null); BYTE_ARRAY_BASE_OFFSET = unsafe.arrayBaseOffset(byte[].class); } catch (Exception e) { throw new RuntimeException(e); } } public static void main(String[] args) { long size = 10; long memoryAddress = unsafe.allocateMemory(size); try { for (int i = 0; i \u0026lt; size; i++) { // Correct bounds unsafe.putByte(memoryAddress + i, (byte) i); } } finally { unsafe.freeMemory(memoryAddress); } } } Here, the loop correctly iterates from 0 to size - 1, preventing the out-of-bounds write.\nUsing java.nio.ByteBuffer # A safer alternative involves using java.nio.ByteBuffer for off-heap memory management. ByteBuffer provides bounds checking, reducing the risk of out-of-bounds writes.\nimport java.nio.ByteBuffer; public class OffHeapByteBufferExample { public static void main(String[] args) { int size = 10; ByteBuffer buffer = ByteBuffer.allocateDirect(size); for (int i = 0; i \u0026lt; size; i++) { // Correct bounds buffer.put((byte) i); } buffer.flip(); // Prepare the buffer for reading while (buffer.hasRemaining()) { System.out.println(buffer.get()); } } } In this example, ByteBuffer manages off-heap memory safely with built-in bounds checking.\nManaging off-heap memory in Java can offer performance benefits but requires careful handling to avoid out-of-bounds writes. Using sun.misc.Unsafe provides powerful capabilities but comes with significant risks. For safer memory management, java.nio.ByteBuffer is recommended due to its automatic bounds checking. Always perform thorough bounds checking and prefer higher-level abstractions to ensure memory safety. But there are changes in newer Java releases.\nHandling Larger Data Structures # For more complex data structures, you can use methods like asIntBuffer, asLongBuffer, etc., which provide views of the buffer with different data types.\nExample with IntBuffer # import java.nio.ByteBuffer; import java.nio.IntBuffer; public class OffHeapIntBufferExample { public static void main(String[] args) { int size = 10; // Allocate off-heap memory ByteBuffer byteBuffer = ByteBuffer.allocateDirect(size * Integer.BYTES); IntBuffer intBuffer = byteBuffer.asIntBuffer(); // Writing data to the buffer for (int i = 0; i \u0026lt; size; i++) { intBuffer.put(i); // Proper bounds checking is inherent } // Preparing the buffer to read the data intBuffer.flip(); // Reading data from the buffer while (intBuffer.hasRemaining()) { System.out.println(intBuffer.get()); } } } Explanation # Allocation : ByteBuffer.allocateDirect(size * Integer.BYTES) allocates a direct buffer with enough space for size integers.\nIntBuffer View : byteBuffer.asIntBuffer() creates an IntBuffer view of the ByteBuffer, directly allowing you to work with integers.\nWriting and Reading : The put and get methods of IntBuffer are used for writing and reading integers, with bounds checking.\nUsing java.nio.ByteBuffer for off-heap memory management in Java provides a safer alternative to sun.misc.Unsafe, ensuring automatic bounds checking and reducing the risk of out-of-bounds writes. Following the outlined examples and strategies, you can effectively manage off-heap memory while maintaining code safety and integrity.\nOffHeap usage in Java 21 # Java 21 introduces several enhancements and features for off-heap memory management, focusing on improved performance, safety, and ease of use. While the core approach using java.nio.ByteBuffer remains prevalent, new features and improvements in the Java ecosystem help with off-heap usage.\nCritical Features for Off-Heap Usage in Java 21 # Foreign Function \u0026amp; Memory API (Preview): Java 21 includes a preview of the Foreign Function \u0026amp; Memory API, designed to facilitate interactions with native code and memory. This API allows for safe and efficient off-heap memory access and manipulation.\nEnhanced ByteBuffer Operations : The ByteBuffer class has been optimised and extended with more operations, making it more efficient for handling off-heap memory.\nMemory Segments and Access VarHandles : Java 21 introduces memory segments and access varhandles for structured memory access. This provides a safer and more structured way to handle off-heap memory.\nForeign Function \u0026amp; Memory API # The Foreign Function \u0026amp; Memory API allows Java programs to interoperate with native libraries and manage off-heap memory more safely and efficiently. This API includes classes such as MemorySegment, MemoryAddress, and MemoryLayout to represent and manipulate memory.\nExample Using Foreign Function \u0026amp; Memory API # Here is an example of how you might use the Foreign Function \u0026amp; Memory API to allocate and manipulate off-heap memory:\nimport jdk.incubator.foreign.*; public class OffHeapExample { public static void main(String[] args) { // Allocate 100 bytes of off-heap memory try (MemorySegment segment = MemorySegment.allocateNative(100)) { MemoryAddress baseAddress = segment.baseAddress(); // Write data to off-heap memory for (int i = 0; i \u0026lt; 100; i++) { segment.set(ValueLayout.JAVA_BYTE, i, (byte) i); } // Read data from off-heap memory for (int i = 0; i \u0026lt; 100; i++) { byte value = segment.get(ValueLayout.JAVA_BYTE, i); System.out.println(\u0026#34;Value at index \u0026#34; + i + \u0026#34;: \u0026#34; + value); } } } } Enhanced ByteBuffer Operations # While ByteBuffer remains a crucial class for off-heap memory management, Java 21\u0026rsquo;s performance improvements make it even more suitable for high-performance applications.\nMemory Segments and Access VarHandles # The new memory segment API provides a safer and more structured way to handle off-heap memory, replacing the need for sun.misc.Unsafe.\nExample Using Memory Segments # import jdk.incubator.foreign.*; public class MemorySegmentExample { public static void main(String[] args) { // Allocate 10 integers worth of off-heap memory try (MemorySegment segment = MemorySegment.allocateNative(10 * 4)) { VarHandle intHandle = MemoryHandles.varHandle(int.class, ByteOrder.nativeOrder()); // Writing data to the memory segment for (int i = 0; i \u0026lt; 10; i++) { intHandle.set(segment.baseAddress().addOffset(i * 4), i); } // Reading data from the memory segment for (int i = 0; i \u0026lt; 10; i++) { int value = (int) intHandle.get(segment.baseAddress().addOffset(i * 4)); System.out.println(\u0026#34;Value at index \u0026#34; + i + \u0026#34;: \u0026#34; + value); } } } } Java 21 enhances off-heap memory management with new and improved features such as the Foreign Function \u0026amp; Memory API and enhanced ByteBuffer operations. These features provide safer, more efficient, and more flexible ways to interact with off-heap memory. By leveraging these new capabilities, developers can write high-performance, memory-safe applications that interact seamlessly with native libraries and off-heap memory.\nDifferences in OffHeap usage between Java 17 and Java 21 # Java 21 introduces several enhancements and new features for off-heap memory management compared to Java 17. Here’s a detailed comparison highlighting the key differences:\nForeign Function \u0026amp; Memory API # Java 17 :\nIncubator Stage : The Foreign Function \u0026amp; Memory API was in an incubator stage, which was experimental and subject to change. The API provided the foundations for off-heap memory access and foreign function calls but lacked maturity and stability.\nJava 21 :\nPreview Stage : The Foreign Function \u0026amp; Memory API is now in a preview stage, indicating it has become more stable and mature. This API allows for safer, more efficient off-heap memory management and interoperability with native libraries.\nMemorySegment and MemoryAddress : These abstractions provide structured and safe access to off-heap memory, reducing the risks associated with manual memory management.\nMemoryLayout and VarHandles : These additions enable developers to describe the layout of memory segments and access them in a type-safe manner using varhandles.\nEnhanced ByteBuffer Operations # Java 17 :\nBasic Operations : The ByteBuffer class in Java 17 supports basic operations for off-heap memory allocation and access through allocateDirect.\nPerformance : While effective, the performance optimisations were less advanced than in Java 21.\nJava 21 :\nPerformance Improvements : Java 21 includes performance optimisations for ByteBuffer, making it more efficient for high-performance applications.\nExtended Operations : Additional operations and methods may have been introduced or optimised to enhance functionality and ease of use.\nMemory Segments and VarHandles # Java 17 :\nLimited Usage : Memory segments and varhandles were part of the incubator Foreign Memory Access API, not widely adopted or stable.\nJava 21 :\nStabilised and Enhanced : These concepts have been further developed and stabilised, providing a more robust way to handle off-heap memory.\nStructured Access : Memory segments offer a structured way to allocate, access, and manage off-heap memory, reducing risks and improving safety.\nVarHandles : Provide a type-safe mechanism to manipulate memory, akin to low-level pointers but with safety checks.\nSummary of Differences # API Maturity : The Foreign Function \u0026amp; Memory API has evolved from an incubator stage in Java 17 to a preview stage in Java 21, providing a more stable and feature-rich environment.\nPerformance : Java 21 offers performance improvements and more optimised operations for ByteBuffer and other memory-related classes.\nMemory Management : Java 21 introduces more structured and safer ways to handle off-heap memory with memory segments and varhandles.\nSafety and Efficiency : Java 21\u0026rsquo;s improvements emphasise safety and efficiency, reducing the risks associated with manual off-heap memory management.\nThese advancements in Java 21 enhance the capabilities and safety of off-heap memory management, making it a more robust choice for developers needing high-performance memory operations.\n","date":"15 May 2024","externalUrl":null,"permalink":"/posts/cwe-787-the-bird-eye-view-for-java-developers/","section":"Posts","summary":"The term “CWE-787: Out-of-bounds Write \" likely refers to a specific security vulnerability or error in software systems. Let’s break down what it means:\nOut-of-bounds Write : This is a type of vulnerability where a program writes data outside the boundaries of pre-allocated fixed-length buffers. This can corrupt data, crash the program, or lead to the execution of malicious code.\n","title":"CWE-787 - The Bird-Eye View for Java Developers","type":"posts"},{"content":"","date":"15 May 2024","externalUrl":null,"permalink":"/tags/include/","section":"Tags","summary":"","title":"Include","type":"tags"},{"content":"","date":"15 May 2024","externalUrl":null,"permalink":"/tags/offheap/","section":"Tags","summary":"","title":"Offheap","type":"tags"},{"content":"","date":"7 May 2024","externalUrl":null,"permalink":"/tags/cybersecurity/","section":"Tags","summary":"","title":"Cybersecurity","type":"tags"},{"content":"","date":"7 May 2024","externalUrl":null,"permalink":"/tags/error-handling/","section":"Tags","summary":"","title":"Error Handling","type":"tags"},{"content":" What is ErrorHandling? # Error handling refers to the programming practice of anticipating, detecting, and responding to exceptions or errors in software during its execution. Errors may occur for various reasons, such as invalid user inputs, hardware failures, or bugs in the code. Proper error handling helps ensure that the program can handle such situations gracefully by resolving the Error, compensating for it, or failing safely.\nWhat is ErrorHandling? How is error handling done in Java? Types of Exceptions Exception Handling Mechanisms Try-Catch Block Throws Keyword Throw Keyword Custom Exceptions Best Practices Why is Error handling necessary for security? How to avoid lousy Error Handling? Information Leakage through Error Messages Denial of Service (DoS) Exception Handling Overhead Uncaught Exceptions Improper Use of Checked and Unchecked Exceptions Security Decisions Based on Exception Handling Error Handling and Code Injection Risks Here are some critical aspects of error handling:\nDetection : Identifying situations that may lead to errors.\nIntervention : Implement strategies to handle the Error once it is detected. This may include logging the Error for further analysis, notifying users or administrators, and attempting recovery.\nRecovery : Trying to continue the program\u0026rsquo;s execution by bypassing the Error or correcting the error condition.\nGraceful Degradation : If recovery is not possible, the system may continue operating with reduced functionality.\nFail-Safe Operations : Ensuring that the system fails in a way that does not cause harm or excessive inconvenience to the user.\nProgramming languages and environments typically provide mechanisms to handle errors, such as Java, Python, or C++ exceptions. Handling these exceptions allows developers to write more robust and reliable software.\nHow is error handling done in Java? # Error handling in Java is primarily accomplished through the use of exceptions. Exceptions are events that can alter the normal flow of program execution when an error or other unusual condition occurs. Java provides a robust and structured way to catch and handle these exceptions, making it possible to manage errors effectively. Here\u0026rsquo;s how error handling is typically done in Java:\nTypes of Exceptions # Java categorises exceptions into two main types:\nChecked Exceptions : These exceptions must be explicitly caught or declared in the method where they might be thrown. They are checked at compile time, meaning the compiler requires the programmer to handle them, typically through try-catch blocks or by declaring the Exception in the method signature with throws.\nUnchecked Exceptions : These include runtime exceptions and errors and do not need to be explicitly handled. Runtime exceptions typically result from programming errors such as logic mistakes, improper use of an API, etc. Errors are used by the Java runtime to indicate serious problems that a reasonable application should not try to catch, like OutOfMemoryError.\nException Handling Mechanisms # Try-Catch Block # The primary exception mechanism is the try-catch block. Here\u0026rsquo;s how it works:\ntry { //code that might throw an exception } catch (ExceptionType name) { //code that handles the Exception } finally { //code that executes after try-catch, regardless of whether an exception was thrown } **try** : The block that contains the code which might throw an exception.\n**catch** : This block catches the Exception thrown by the try block. Multiple catch blocks can handle different exceptions.\n**finally** : This block is optional and executes after the try-and-catch blocks. It executes regardless of whether an exception was thrown or caught. It\u0026rsquo;s typically used for cleanup activities (like closing file streams).\nThrows Keyword # If a method is capable of causing an exception that it does not handle, it must declare this Exception with the throws keyword in its method signature:\npublic void myMethod() throws IOException { //code that might throw an IOException } Throw Keyword # Java also allows you to throw an exception explicitly using the throw keyword. This is often used to throw a custom exception or re-throw a caught exception after some specific handling:\n**throw new Exception(\u0026#34;This is an error message\u0026#34;);** Custom Exceptions # You can create your exception types by extending the Exception class (for checked exceptions) or the RuntimeException class (for unchecked exceptions). This is useful for creating application-specific exceptions that can carry more contextual information about the Error:\npublic class MyCustomException extends Exception { public MyCustomException(String message) { super(message); } } Best Practices # - Catch the most specific Exception first before the more general ones.\n- Avoid catching Throwable, Exception, or RuntimeException unless necessary, as it can hide bugs.\n- Always clean up after handling an exception, either in a finally block or using try-with-resources for AutoCloseable resources.\nBy following these structured approaches, Java programs can handle errors gracefully, making applications more robust, secure, and user-friendly.\nWhy is Error handling necessary for security? # Error handling is critically important for security for several reasons, as it directly impacts the robustness and resilience of a system. Here are some key ways in which effective error handling enhances security:\nPreventing Information Disclosure : Poor error handling can unintentionally leak sensitive information. For example, a detailed error message could reveal system details, such as file paths, database schema, or software versions, which attackers could exploit. Proper error handling should mask or generalise error messages shown to users while logging detailed information securely for debugging purposes by administrators.\nAvoiding Denial of Service (DoS) : Software that does not gracefully handle errors can crash or become unresponsive when faced with unexpected conditions, making it vulnerable to DoS attacks. Proper error handling ensures that the application can continue to operate or fail safely without affecting overall availability.\nMitigating Injection Flaws : Applications that fail to handle errors properly are often vulnerable to injection attacks, such as SQL injection, where attackers exploit error messages to glean insights about the database structure. By securely handling errors, applications can prevent such attacks from being successful.\nEnsuring Data Integrity : Effective error handling can help maintain data integrity by checking for errors during data processing and preventing corrupt or partial data from being saved to the database. This is crucial in avoiding scenarios where erroneous data could lead to security vulnerabilities or functional errors.\nManaging Untrusted Data and Inputs : By handling errors properly, applications can safely deal with untrusted inputs that might otherwise cause failures or security breaches. For instance, if an application tries to process a considerable file and fails due to resource limitations, properly handling this condition prevents it from being used as a vector for attacks.\nUpholding Compliance and Trust : Many regulatory standards require robust error handling as part of compliance mandates. Failing to implement proper error handling can lead to compliance failures and legal liabilities. Moreover, a well-handled error scenario can maintain user trust, whereas repeated failures or uninformative error messages can erode it.\nOverall, robust error handling is essential in designing secure systems. It prevents specific security vulnerabilities and contributes to the software\u0026rsquo;s overall resilience and reliability.\nHow to avoid lousy Error Handling? # When not appropriately implemented, error handling in Java can have several security implications. Here\u0026rsquo;s a detailed look at potential issues and how to mitigate them:\nInformation Leakage through Error Messages # Problem : Detailed error messages can inadvertently reveal system valuable information to attackers. For example, stack traces exposed to users might include information about the software architecture, third-party libraries being used, database information, or specific methods in the code.\nMitigation : Avoid sending detailed error messages to the client or end-user. Instead, log these details internally, where developers or system administrators can use them for debugging. Show users generic, user-friendly error messages that do not disclose sensitive information.\nDenial of Service (DoS) # Problem : If exceptions are not handled properly, an attacker might be able to exploit this by sending requests that they know will cause exceptions, potentially leading the application to crash or become unresponsive.\nMitigation : Implement robust Exception handling that isolates error conditions and ensures the application can recover gracefully. Use techniques like input validation to prevent problematic data from causing unexpected behaviour.\nException Handling Overhead # Problem : Creating, throwing, and catching exceptions can be resource-intensive, especially if the exceptions involve stack trace generation. If an application generates many exceptions as part of normal operations, it can lead to performance issues and resource exhaustion.\nMitigation : Optimise exceptions by avoiding them in standard flow control and using them only for exceptional conditions. Consider using return codes or validation methods to handle common, expected conditions.\nUncaught Exceptions # Problem : The application can miss exceptions, causing the program to terminate unexpectedly, leading to potential denial of service and other stability issues.\nMitigation : Ensure that all parts of the application are covered by adequate exception handling, and use global exception handlers as a last resort to catch and log any unhandled exceptions.\nImproper Use of Checked and Unchecked Exceptions # Problem : Misusing checked and unchecked exceptions can lead to poorly designed code that hides errors (unchecked) or burdens the developer with too many error-handling responsibilities (checked), potentially leading to ignored exceptions.\nMitigation : Use checked exceptions for recoverable conditions and where you want to enforce the caller\u0026rsquo;s handling. Use unchecked exceptions for programming errors that should not be handled programmatically.\nSecurity Decisions Based on Exception Handling # Problem : Relying on exception handling to make security decisions, such as authentication or validation, can lead to logic errors and potential security vulnerabilities.\nMitigation : Design security controls independently of the exception-handling mechanism. Ensure that security decisions are made based on secure, well-established practices rather than the presence or absence of exceptions.\nError Handling and Code Injection Risks # Problem : Poorly handled errors can lead to injection vulnerabilities if the output from exceptions is not properly sanitised before being displayed to the user.\nMitigation : Always sanitise any dynamic content displayed to users, including error messages, to prevent cross-site scripting (XSS) and other injection attacks.\nDevelopers can significantly enhance Java applications\u0026rsquo; security posture by understanding and addressing these security implications. Effective error handling prevents crashes, improves user experience, and strengthens the application\u0026rsquo;s resistance to malicious attacks.\n","date":"7 May 2024","externalUrl":null,"permalink":"/posts/mastering-secure-error-handling-in-java-best-practices-and-strategies/","section":"Posts","summary":"What is ErrorHandling? # Error handling refers to the programming practice of anticipating, detecting, and responding to exceptions or errors in software during its execution. Errors may occur for various reasons, such as invalid user inputs, hardware failures, or bugs in the code. Proper error handling helps ensure that the program can handle such situations gracefully by resolving the Error, compensating for it, or failing safely.\n","title":"Mastering Secure Error Handling in Java: Best Practices and Strategies","type":"posts"},{"content":"Logging is essential to software development, recording information about the software\u0026rsquo;s operation. This can help developers understand the system\u0026rsquo;s behaviour, troubleshoot issues, and monitor the system in production. Here\u0026rsquo;s a basic overview of logging in software development:\nPurpose of Logging Debugging and Troubleshooting: Monitoring System Health: Audit Trails: Security Analysis: Operational Intelligence: Regulatory Compliance: Notification of Important Events: What to Log Errors and Exceptions System Events User Actions Performance Metrics Security Events Warnings Logging Levels DEBUG INFO NOTICE WARNING ERROR CRITICAL ALERT EMERGENCY Choosing the Right Level Logging Best Practices Use Established Logging Frameworks Implement Appropriate Log Levels Ensure Logs Are Contextual and Informative Centralise Log Management Secure and Protect Log Data Regularly Monitor and Analyse Logs Manage Log Volume Automate Log Analysis Implement Log Retention Policies Use Asynchronous and Non-blocking Logging Standardise Logs Across Services Tools for Log Management ELK Stack Splunk Graylog Fluentd Datadog Loggly Papertrail Prometheus and Grafana Choosing the Right Tool Common Pitfalls Over-Logging Under-Logging Ignoring Logs Not Protecting Log Information Inconsistent Logging Lack of Context in Logs Blocking or Synchronous Logging Poor Log Management and Retention Practices Failing to Plan for Log Scalability Purpose of Logging # The purpose of logging in software development is multifaceted, encompassing several vital aspects that collectively improve software systems\u0026rsquo; operability, security, and maintainability. Here are the primary purposes of logging:\nDebugging and Troubleshooting: # Logging gives developers detailed insights into the application\u0026rsquo;s behaviour, allowing them to trace the steps leading up to errors, exceptions, and other anomalous behaviours. This is invaluable for debugging and can significantly reduce the time needed to diagnose and fix issues.\nMonitoring System Health: # Logs can provide real-time information about the health of an application, including performance metrics such as response times, throughput, and resource utilisation. This helps proactively manage system performance and identify potential issues before they affect users.\nAudit Trails: # Logging creates an audit trail in many applications, particularly those that handle financial transactions, sensitive data, or personal information. This helps track user actions and system changes, which is crucial for compliance with regulations and standards (such as GDPR, HIPAA, or SOX).\nSecurity Analysis: # Logs can detect and alert potential security incidents. For instance, a high number of failed login attempts could indicate a brute-force attack. Logs also help in the forensic analysis after an incident, enabling the understanding of how the security breach occurred and the extent of the impact.\nOperational Intelligence: # Analysing logs, businesses can gain insights into user behaviour, system usage patterns, and operational bottlenecks. This information can guide decisions on system improvements, user support, and new feature development.\nRegulatory Compliance: # Logging is not just an operational tool but a legal requirement for many industries. Logs must be maintained to demonstrate compliance with various regulatory frameworks and show that the system performs correctly and securely.\nNotification of Important Events: # Logging systems can be configured to send alerts when critical system events occur, such as system outages, significant performance degradation, or other critical issues that require immediate attention.\nLogging is an essential discipline in software development that enhances visibility into applications, ensuring they run smoothly, securely, and in compliance with legal and operational requirements. It also helps developers and IT professionals effectively manage, diagnose, and optimise their software environments.\nWhat to Log # Deciding what to log in a software system is crucial for ensuring you gather enough information to be helpful in debugging, monitoring, and compliance without overwhelming the system or the teams that need to parse through the logs. Here are key categories and specific elements you should consider logging:\nErrors and Exceptions # Critical Failures : Any errors that cause a part of your system to fail or potentially disrupt service.\nExceptions : Catch and log exceptions with stack traces and context to understand why they occurred.\nSystem Events # Startups and Shutdowns : Record when your system or service starts and stops.\nConfiguration Changes : Log any changes in a system configuration that might affect behaviour or performance.\nScheduled Tasks : Record when scheduled tasks begin and end, especially if they\u0026rsquo;re crucial to system functionality.\nUser Actions # Logins and Logouts : Tracking user sessions can help detect unauthorised access and understand user engagement.\nImportant Transactions : Logging these activities is vital, especially in systems handling payments, orders, or sensitive operations.\nAccess to Sensitive Data : Log in when users access sensitive information to comply with privacy laws and regulations.\nPerformance Metrics # Response Times : Log how long it takes to respond to user requests.\nSystem Utilisation : Include metrics on CPU, memory, disk, and network usage.\nService Availability : Record any downtime or interruptions in service.\nSecurity Events # Failed Logins : An excessive number of failed login attempts can indicate a brute-force attack.\nPermission Changes : Track user permissions changes, especially for administrative access users.\nSecurity Breaches : Log any detected breaches or potential security threats.\nWarnings # Resource Limits : Warnings when resources (e.g., memory, disk space) run low.\nDeprecations : Log usage of deprecated APIs or features that will be removed in future.\nLogging Levels # Log levels are a way to categorise entries in your logs based on their importance and the detail of information they provide. These levels help filter logs, like debugging, monitoring, and alerting. Here\u0026rsquo;s a breakdown of expected log levels used in many logging systems, from the most verbose to the least:\nDEBUG # Purpose : This is the most detailed level for total diagnostic output. It includes valuable information for developers during development and debugging to understand precisely what the system is doing.\nUse Case : You would enable DEBUG logging when trying to solve a tricky bug or when you need a detailed trace of how data flows through your system.\nINFO # Purpose : General information about the application\u0026rsquo;s operation. These entries should be informative and relevant to users or system administrators monitoring the healthy functioning of the application.\nUse Case : Routine operations like user logins, SQL logs, and other operational milestones.\nNOTICE # Purpose : Important runtime events that are not necessarily errors but may require attention or be significant in system auditing or analysis.\nUse Case : Deprecation warnings, minor configuration issues, or other messages that don\u0026rsquo;t require immediate action but should be noted for potential future relevance.\nWARNING # Purpose : Indicative of something unexpected or a potential problem in the future. However, the application can continue running.\nUse Case : Recoverable malfunctions, such as retrying operations, missing secondary data, or running with default values due to missing configuration.\nERROR # Purpose : This indicator indicates issues that are of immediate concern, as they may hinder system operations or result in a partial application failure.\nUse Case : Runtime errors, inability to access a necessary resource, exceptions that are handled but disrupt regular operation.\nCRITICAL # Purpose : Very serious issues that might cause the application to terminate or affect essential functionality.\nUse Case : Critical conditions include data loss scenarios, out-of-memory, or data corruption.\nALERT # Purpose : A step above CRITICAL, requiring immediate attention. This typically involves issues that need to be fixed immediately to prevent stoppage or significant damage to operations.\nUse Case : Breaches in security, system components going down, loss of connectivity.\nEMERGENCY # Purpose : This is the highest level, used when the system is unusable or major parts of the system are non-functional.\nUse Case : Complete system outage, catastrophic failure, or anything that requires immediate and urgent attention to prevent harm or significant disruption.\nChoosing the Right Level # When choosing which level to log in, consider the following:\nThe audience for the logs : Developers, system administrators, end-users, or auditors.\nThe environment : Development, testing, staging, or production.\nThe severity of the event : The impact on the system\u0026rsquo;s functionality and the user\u0026rsquo;s experience.\nProper use of log levels allows you to control the verbosity of logging output, making the logs both manageable and meaningful. This control is significant in production environments where excessive logging can lead to performance degradation and increased log storage and management costs.\nLogging Best Practices # Adopting best practices in logging is crucial for creating a robust and effective logging strategy. Here are several fundamental guidelines to help ensure your logging system is both efficient and valuable:\nUse Established Logging Frameworks # Why : Leverage well-maintained and community-tested libraries to handle routine logging tasks. Examples include Log4j for Java, Serilog for .NET, Winston for Node.js, and Python\u0026rsquo;s built-in logging module.\nBenefit : These frameworks support features like log rotation, different logging levels, and integration with various logging backends.\nImplement Appropriate Log Levels # Why : Differentiate log messages according to severity levels (DEBUG, INFO, WARNING, ERROR, etc.).\nBenefit : Allows fine-grained control over which log entries are output based on the current runtime environment, helping to reduce noise in production logs and focus on relevant data.\nEnsure Logs Are Contextual and Informative # Why : Log messages should include enough context to be understood independently. Context can consist of timestamps, user IDs, session IDs, and other relevant details.\nBenefit : Makes troubleshooting easier by clarifying when and where events occur, especially in distributed systems.\nCentralise Log Management # Why : In distributed systems, consolidate logs from multiple sources into a central log management solution.\nBenefit : Simplifies monitoring and analysis, enabling more effective incident detection and response. Tools like ELK Stack, Splunk, and Graylog are popular choices.\nSecure and Protect Log Data # Why : Logs often contain sensitive information which must be protected.\nBenefit : Prevents sensitive data exposure and complies with data protection regulations (like GDPR).\nRegularly Monitor and Analyse Logs # Why : Active log monitoring helps identify and respond to issues promptly.\nBenefit : Reduces system downtime and can alert to emerging issues before they become critical.\nManage Log Volume # Why : Avoid logging too much unnecessary information.\nBenefit : It helps manage log sizes, reduces storage costs, and improves log readability.\nAutomate Log Analysis # Why : Use tools and scripts to analyse logs for patterns and anomalies automatically.\nBenefit : Enhances the ability to quickly identify issues without manually sifting through logs, especially in large-scale systems.\nImplement Log Retention Policies # Why : Define how long logs should be retained based on the type of data and compliance requirements.\nBenefit : It ensures that logs are available for sufficient time for audits and analysis while avoiding unnecessary storage costs.\nUse Asynchronous and Non-blocking Logging # Why : Prevent logging operations from impacting application performance.\nBenefit : Minimises the performance overhead of logging activities on the main application processes.\nStandardise Logs Across Services # Why : Use a consistent format across different services and parts of your application.\nBenefit : It simplifies the analysis and correlation of logs from different sources, which is particularly valuable in microservices architectures.\nBy integrating these best practices into your development and operational processes, you can maximise the benefits of logging, making it a powerful tool for maintaining and improving your software systems\u0026rsquo; performance, reliability, and security.\nTools for Log Management # Effective log management is critical for maintaining system performance, ensuring security, and troubleshooting issues across software applications. Here are some of the most popular tools and platforms that help in aggregating, analysing, and managing logs:\nELK Stack # Components : Elasticsearch, Logstash, and Kibana.\nUse Case : ELK Stack is one of the most popular open-source choices for log management. Elasticsearch acts as a search and analytics engine, Logstash is used for log ingestion and processing, and Kibana is used for visualisation and querying.\nBenefits : Highly customisable and scalable, capable of handling massive volumes of data.\nSplunk # Use Case : Splunk is a powerful commercial solution with a web-based interface that provides extensive capabilities for searching, monitoring, and analysing machine-generated data.\nBenefits : Known for its advanced analytics features and extensive out-of-the-box functionalities that can handle complex queries across large datasets.\nGraylog # Use Case : Graylog is an open-source log management tool that offers centralised log management, efficient analysis, and a user-friendly dashboard.\nBenefits : It\u0026rsquo;s known for its simplicity and efficiency in storing, searching, and analysing large amounts of data.\nFluentd # Use Case : Fluentd is an open-source data collector for unified logging layers, which allows you to unify data collection and consumption for better use and understanding of data.\nBenefits : Fluentd is particularly noted for its flexibility and a wide array of plugins that integrate with many data sources and output formats.\nDatadog # Use Case : Datadog provides cloud-scale monitoring that includes the ability to collect, search, and analyse log data, as well as infrastructure and application performance monitoring.\nBenefits : It offers real-time logs, sophisticated alerting, and seamless integration with various cloud services.\nLoggly # Use Case : Loggly provides cloud-based log management services that can analyse large volumes of data and extract useful information, typically geared towards enterprise-level users.\nBenefits : Loggly is easy to set up and integrates well with existing applications, providing robust search capabilities and interactive visualisations.\nPapertrail # Use Case : Papertrail offers cloud-based log management that focuses on simplicity and fast setup, providing instant log visibility and analysis.\nBenefits : Its simplicity makes it ideal for smaller applications or teams needing quick setup without extensive configuration.\nPrometheus and Grafana # Use Case : While Prometheus is primarily used for monitoring and alerting, it can be combined with Grafana for log visualisation. This combo is often used for monitoring and visually analysing metrics.\nBenefits : Open-source, decisive for visualising trends and patterns, and highly extensible with Grafana\u0026rsquo;s advanced dashboard capabilities.\nChoosing the Right Tool # When selecting a log management tool, consider factors like:\nVolume of Data : The amount of log data you generate.\nThe complexity of the Environment : Whether you are managing logs from a single application or distributed systems.\nBudget : Open-source vs. commercial solutions.\nIntegration Needs : Compatibility with existing tools and platforms in your ecosystem.\nCompliance Requirements : Certain industries might need specific features to comply with legal standards.\nEach tool has strengths and is suitable for different organisational needs and sizes. The right choice will depend on your specific requirements, including scalability, ease of use, budget, and the specific features you need.\nCommon Pitfalls # Logging is a powerful tool in software development, but it can also lead to issues if not appropriately managed. Here are some common pitfalls associated with logging and how to avoid them:\nOver-Logging # Problem : Logging too much information can clutter log files, make them challenging to manage, and lead to performance degradation.\nSolution : Use appropriate log levels to control the verbosity of the logs. Log only what is necessary for troubleshooting and operational monitoring. Regularly review and adjust what is being logged based on current needs.\nUnder-Logging # Problem : Insufficient logging can leave you without enough information to diagnose issues, especially in production environments.\nSolution : Ensure critical paths and user transactions are well-logged. Log errors effectively, including error handling and catching exceptions. Use structured logging to enhance the quality and usefulness of the logs.\nIgnoring Logs # Problem : Not regularly reviewing logs can lead to missed opportunities to detect or fix issues early.\nSolution : Implement log monitoring and alerting tools to watch for unusual activity or errors actively. Establish routines for checking logs, especially after deployments or changes.\nNot Protecting Log Information # Problem : Logs can contain sensitive information, which, if exposed, can lead to security breaches.\nSolution : Secure log files by restricting access and using encryption where necessary. Be mindful of what is logged, especially avoiding logging sensitive user data like passwords or personal information.\nInconsistent Logging # Problem : Inconsistent log formats across different parts of an application can make it difficult to correlate events when analysing logs.\nSolution : Standardise log formats across the application. Use structured logging formats like JSON to make logs more uniform and more straightforward to analyse.\nLack of Context in Logs # Problem : In complex or distributed systems, logs with sufficient context can be easier to interpret.\nSolution : Logs should include contextual information such as timestamps, user IDs, session IDs, and request IDs to clarify the events being logged.\nBlocking or Synchronous Logging # Problem : Synchronous logging can negatively impact application performance, causing delays in user-facing operations.\nSolution : Use asynchronous logging mechanisms to minimise the impact on application performance and ensure logging does not block critical application workflows.\nPoor Log Management and Retention Practices # Problem : Inadequate log rotation and retention policies can lead to manageable log files and increased storage costs.\nSolution : Implement log rotation policies and configure appropriate log retention durations based on business needs and compliance requirements.\nFailing to Plan for Log Scalability # Problem : As systems grow, the volume of logs can increase dramatically, leading to scalability issues.\nSolution : Plan for log scalability from the start. Consider how logs will be handled, stored, and analysed as data volumes grow.\nBy being aware of these common pitfalls and actively working to avoid them, you can ensure that your logging practices enhance rather than hinder your application\u0026rsquo;s development and operation. Effective logging practices lead to more maintainable, reliable, and secure software systems.\n","date":"6 May 2024","externalUrl":null,"permalink":"/posts/decoding-the-logs-essential-insights-for-effective-software-debugging/","section":"Posts","summary":"Logging is essential to software development, recording information about the software’s operation. This can help developers understand the system’s behaviour, troubleshoot issues, and monitor the system in production. Here’s a basic overview of logging in software development:\n","title":"Decoding the Logs: Essential Insights for Effective Software Debugging","type":"posts"},{"content":"","date":"6 May 2024","externalUrl":null,"permalink":"/tags/logging/","section":"Tags","summary":"","title":"Logging","type":"tags"},{"content":"","date":"3 May 2024","externalUrl":null,"permalink":"/tags/access-control/","section":"Tags","summary":"","title":"Access Control","type":"tags"},{"content":"Access control is a security measure that determines who can access resources or perform actions within a system. It involves defining and enforcing policies restricting unauthorised access while allowing authorised users to perform their intended tasks. Access control mechanisms are commonly used in various domains, including computer systems, buildings, and physical assets.\nWhat are the different Access Control ways in software? Discretionary Access Control (DAC): Mandatory Access Control (MAC): Role-Based Access Control (RBAC): Attribute-Based Access Control (ABAC): Rule-Based Access Control (RBAC): What are the secure coding practices in Java for Access Control? Use the Principle of Least Privilege: Implement Authentication and Authorization: Avoid Hardcoding Sensitive Information: Securely Manage Credentials: Secure Communication: Sanitize Inputs: Implement Access Control Checks: Secure Error Handling: Regularly Update Dependencies: Security Testing: What are the pros and cons of Access Control implemented in Java\u0026rsquo;s static or dynamic semantics? Static Semantics: Pros: Cons: Dynamic Semantics: Pros: Cons: What are common Design Patterns in Java for implementing Access Control? Proxy Pattern: Decorator Pattern: Chain of Responsibility Pattern: Strategy Pattern: Facade Pattern: Observer Pattern: Singleton Pattern: Command Pattern: An example in Java for Access Control using a Proxy Step 1: Define the Document Interface Step 2: Implement the Document Class Step 3: Create the Proxy Class Step 4: Demonstrate the Proxy in Action In computer systems, access control typically involves the following components:\nIdentification : Users must identify themselves to the system by providing a username or other unique identifier.\nAuthentication : After identification, users must authenticate themselves by providing credentials such as passwords, biometric data, or security tokens.\nAuthorisation : Once authenticated, users are granted access rights based on their identity and the permissions associated with that identity. Authorisation mechanisms define what resources or actions a user is allowed to access.\nEnforcement : The system enforces access control policies to ensure that only authorised users can access resources or perform specific actions. This may involve mechanisms such as encryption, access control lists (ACLs), or role-based access control (RBAC).\nAccess control can be implemented at various levels, including:\nPhysical access control refers to restricting access to physical locations such as buildings, rooms, or data centres using mechanisms such as locks, keycards, or biometric scanners.\nLogical access control : Managing access to digital resources such as files, databases, or networks through software-based mechanisms such as user accounts, permissions, and encryption.\nAccess control is fundamental to information security and is essential for protecting sensitive data, preventing unauthorised activities, and ensuring compliance with regulations and organisational policies. It helps organisations manage risks associated with unauthorised access and maintain their resources\u0026rsquo; confidentiality, integrity, and availability.\nWhat are the different Access Control ways in software? # In software systems, access control mechanisms are implemented to regulate and manage user access to resources. There are several approaches to access control in software:\nDiscretionary Access Control (DAC): # DAC allows the owner of a resource to control access to that resource. Owners can grant or revoke access permissions to other users or groups. Access control lists (ACLs) and file permissions are typical implementations of DAC. For example, Unix-like operating systems use file permissions (read, write, execute) to control access to files and directories.\nMandatory Access Control (MAC): # MAC enforces access control based on predefined security policies determined by system administrators or security policies. Users do not have control over access rights. Security labels or security clearances are assigned to resources and users. Access decisions are based on the resource\u0026rsquo;s sensitivity and the user\u0026rsquo;s clearance level. SELinux (Security-Enhanced Linux) and TrustedBSD are examples of MAC systems.\nRole-Based Access Control (RBAC): # RBAC assigns permissions to roles, and users are assigned to specific roles. Users inherit the permissions associated with their roles. Roles represent job functions or responsibilities within an organisation, and access decisions are based on user roles. RBAC simplifies administration by centralising access control management and reducing the complexity of managing individual user permissions.\nAttribute-Based Access Control (ABAC): # ABAC evaluates access decisions based on user, resource, and environment attributes. Attributes can include:\nUser attributes (e.g., role, department). Resource attributes (e.g., sensitivity, type). Environmental attributes (e.g., time, location). Policies define conditions that must be met for access to be granted based on attribute values. ABAC provides a flexible and dynamic access control model that can adapt to changing organisational requirements and contexts.\nRule-Based Access Control (RBAC): # Rule-based access control defines access rules or policies that specify conditions under which access is granted or denied. Access decisions are based on evaluating these rules against the attributes of the user, resource, and environment. Rules can be defined using logical expressions, conditions, or patterns.\nThese access control models can be combined or extended to meet the specific security requirements of software systems. The choice of access control mechanism depends on factors such as the system\u0026rsquo;s nature, the data\u0026rsquo;s sensitivity, and organisational policies and regulations.\nWhat are the secure coding practices in Java for Access Control? # Secure coding practices in Java for access control help developers ensure that their applications enforce proper access control policies to protect sensitive data and prevent unauthorised access. Here are some best practices:\nUse the Principle of Least Privilege: # Grant users and components only the permissions they need to perform their tasks and no more. Limit access to sensitive resources to minimise the potential impact of a security breach.\nImplement Authentication and Authorization: # Authenticate users before granting access to protected resources. Use robust authentication mechanisms such as username/password, multi-factor authentication, or OAuth. Implement authorisation mechanisms such as role-based access control (RBAC) or attribute-based access control (ABAC) to enforce access control policies based on user roles, permissions, or attributes.\nAvoid Hardcoding Sensitive Information: # Avoid hardcoding sensitive information such as passwords, API keys, or cryptographic keys directly into the source code. Instead, store them securely in configuration files, environment variables, or secure key management systems.\nSecurely Manage Credentials: # Use secure storage mechanisms such as Java KeyStore (JKS) or Key Management Service (KMS) to store and manage sensitive credentials. Avoid storing plaintext passwords or sensitive information in memory longer than necessary. Use secure storage and encryption techniques when handling sensitive data.\nSecure Communication: # Use secure communication protocols such as HTTPS (SSL/TLS) to encrypt data transmitted between clients and servers. Verify server certificates to prevent man-in-the-middle attacks and ensure the integrity and authenticity of communication.\nSanitize Inputs: # Validate and sanitise user inputs to prevent injection attacks such as SQL injection, Cross-Site Scripting (XSS), or Command Injection. Input validation libraries or frameworks such as OWASP ESAPI or Hibernate Validator are used to validate and sanitise user inputs.\nImplement Access Control Checks: # Perform access control checks at the entry points of sensitive operations or resources. Use access control mechanisms provided by Java frameworks or libraries such as Spring Security or Apache Shiro to enforce access control policies consistently across the application.\nSecure Error Handling: # Implement proper error handling mechanisms to avoid exposing sensitive information in error messages. Use custom error messages or error logging mechanisms to provide informative error messages to developers while hiding sensitive details from end-users.\nRegularly Update Dependencies: # Keep your Java libraries and dependencies up-to-date to ensure that you are using the latest security patches and fixes. Monitor security advisories and vulnerability databases for known security issues in third-party libraries and promptly apply patches or updates.\nSecurity Testing: # Perform regular security testing, including penetration testing, code reviews, and security scans, to identify and remediate security vulnerabilities in your Java applications. Use automated security testing tools such as OWASP ZAP, SonarQube, or FindBugs to identify common security issues and vulnerabilities in your code.\nBy following these secure coding practices, Java developers can strengthen the access control mechanisms in their applications and reduce the risk of security breaches and unauthorised access to sensitive data.\nWhat are the pros and cons of Access Control implemented in Java\u0026rsquo;s static or dynamic semantics? # Access control can be implemented in both Java\u0026rsquo;s static and dynamic semantics. Here are the pros and cons of each approach:\nStatic Semantics: # Pros: # Early Detection of Errors : Static access control mechanisms are enforced at compile-time, allowing for early detection of access control violations. This helps identify potential security vulnerabilities before the code is executed.\nCompile-Time Optimization : The compiler can optimise static access control checks, improving the application\u0026rsquo;s performance by reducing runtime overhead.\nEnforcement of Policies : Static access control mechanisms enforce access control policies uniformly across the entire codebase, ensuring consistency and reducing the likelihood of human error.\nCons: # Limited Flexibility : Static access control mechanisms may lack the flexibility to adapt to runtime changes or dynamic conditions. This can be problematic when access control policies must be dynamically modified or customised based on runtime context.\nLimited Granularity : Static access control mechanisms may have limitations in providing fine-grained access control, especially in complex or dynamic systems where access privileges must be defined at a more granular level.\nDynamic Semantics: # Pros: # Flexibility : Dynamic access control mechanisms can adapt to changing runtime conditions, allowing for more flexible access control policies that can be adjusted based on dynamic factors such as user roles, permissions, or environmental context.\nFine-Grained Access Control : Dynamic access control mechanisms can provide finer-grained access control, allowing for more precise specification of access privileges based on specific runtime conditions or attributes.\nCons: # Runtime Overhead : Dynamic access control checks incur runtime overhead, as access control decisions are made during program execution. This can potentially impact the application\u0026rsquo;s performance, especially in performance-sensitive systems.\nLate Detection of Errors : Dynamic access control mechanisms may only detect access control violations during runtime, which can delay the detection and resolution of security vulnerabilities. If access control policies are not adequately enforced, this increases the risk of security breaches.\nComplexity : Dynamic access control mechanisms can introduce complexity, especially in large or distributed systems, making managing and maintaining access control policies challenging.\nIn summary, the choice between static and dynamic access control semantics in Java depends on factors such as the specific requirements of the application, the level of flexibility needed, and the trade-offs between performance and security. While static access control provides early detection of errors and compile-time optimisation, dynamic access control offers flexibility and fine-grained control but may incur runtime overhead and complexity.\nWhat are common Design Patterns in Java for implementing Access Control? # Several design patterns can be applied in Java to implement access control effectively. Here are some common design patterns used for access control:\nProxy Pattern: # The Proxy pattern allows for controlling access to objects by acting as intermediaries. Proxy objects can enforce access control rules before allowing the client to access the object. Proxies can restrict access to sensitive resources based on user permissions or roles in access control.\nDecorator Pattern: # The Decorator pattern enables dynamic augmentation of object behaviour by wrapping them with additional functionality. In access control, decorators can add security checks or authorisation logic around methods that access sensitive resources.\nChain of Responsibility Pattern: # The Chain of Responsibility pattern creates a chain of handler objects, each responsible for processing a request. The request is passed along the chain until it is handled. In access control, a chain of responsibility can be used to enforce access control checks at various application levels, with each handler responsible for verifying specific access rights.\nStrategy Pattern: # The Strategy pattern defines a family of algorithms, encapsulates each one, and makes them interchangeable. Clients can choose the appropriate algorithm at runtime. In access control, the strategy pattern can be used to implement different access control strategies (e.g., RBAC, ABAC) and dynamically allow the application to switch between them.\nFacade Pattern: # The Facade pattern provides a unified interface to a set of interfaces in a subsystem, making it easier to use. In access control, a facade can hide the complexity of access control mechanisms by providing a simplified interface for clients to interact with while internally handling the access control logic.\nObserver Pattern: # The Observer pattern defines a one-to-many dependency between objects, where the state changes in one object are propagated to its dependents. In access control, observers can notify interested parties (e.g., audit logs, monitoring systems) about access control events, such as access granted or denied.\nSingleton Pattern: # The Singleton pattern ensures that a class has only one instance and provides a global access point to that instance. In access control, a singleton can be used to manage a centralised access control policy or configuration, ensuring consistency across the application.\nCommand Pattern: # The Command pattern encapsulates a request as an object, thereby allowing for parameterisation of clients with queues, requests, and operations. In access control, commands can represent access control requests, encapsulating the necessary information (e.g., user, resource, action) to perform access control checks.\nBy applying these design patterns, developers can design flexible, maintainable, and extensible access control mechanisms in Java applications, ensuring that access to sensitive resources is adequately managed and enforced.\nAn example in Java for Access Control using a Proxy # To illustrate access control using a proxy in Java, let\u0026rsquo;s create a simple example where we have a Document class that users can read or write. We will then use a DocumentProxy class to control access to the Document based on the user\u0026rsquo;s role (e.g., \u0026ldquo;admin\u0026rdquo; can read and write, while \u0026ldquo;user\u0026rdquo; can only read).\nStep 1: Define the Document Interface # This interface declares the methods for interacting with a document.\npublic interface Document { void displayDocument(); void modifyDocument(String content); } Step 2: Implement the Document Class # This class implements the Document interface. It represents a real document that can be modified and displayed.\npublic class RealDocument implements Document { private String content; public RealDocument(String content) { this.content = content; } @Override public void displayDocument() { System.out.println(\u0026#34;Displaying Document: \u0026#34; + content); } @Override public void modifyDocument(String content) { this.content = content; System.out.println(\u0026#34;Document Modified\u0026#34;); } } Step 3: Create the Proxy Class # The DocumentProxy class controls access to the RealDocument based on the user\u0026rsquo;s role.\npublic class DocumentProxy implements Document { private RealDocument realDocument; private String userRole; public DocumentProxy(RealDocument realDocument, String userRole) { this.realDocument = realDocument; this.userRole = userRole; } @Override public void displayDocument() { realDocument.displayDocument(); } @Override public void modifyDocument(String content) { if (\u0026#34;admin\u0026#34;.equals(userRole)) { realDocument.modifyDocument(content); } else { System.out.println(\u0026#34;Access Denied: You do not have permission to modify the document.\u0026#34;); } } } Step 4: Demonstrate the Proxy in Action # Finally, let\u0026rsquo;s create a simple demo to illustrate how the proxy controls access based on the user role.\npublic class ProxyDemo { public static void main(String[] args) { RealDocument document = new RealDocument(\u0026#34;Original Content\u0026#34;); //user trying to modify the document DocumentProxy userProxy = new DocumentProxy(document, \u0026#34;user\u0026#34;); userProxy.displayDocument(); userProxy.modifyDocument(\u0026#34;New User Content\u0026#34;); // Admin trying to modify the document DocumentProxy adminProxy = new DocumentProxy(document, \u0026#34;admin\u0026#34;); adminProxy.displayDocument(); adminProxy.modifyDocument(\u0026#34;New Admin Content\u0026#34;); adminProxy.displayDocument(); } } In this example, when a \u0026ldquo;user \u0026quot; tries to modify the document through the \u0026ldquo;DocumentProxy \u0026ldquo;, access is denied. However, when an \u0026ldquo;admin \u0026quot; attempts to change it, the operation is successful. This demonstrates how a proxy can be used to control access to certain functionalities based on user roles.\n","date":"3 May 2024","externalUrl":null,"permalink":"/posts/secure-coding-practices-access-control/","section":"Posts","summary":"Access control is a security measure that determines who can access resources or perform actions within a system. It involves defining and enforcing policies restricting unauthorised access while allowing authorised users to perform their intended tasks. Access control mechanisms are commonly used in various domains, including computer systems, buildings, and physical assets.\n","title":"Secure Coding Practices - Access Control","type":"posts"},{"content":"","date":"22 April 2024","externalUrl":null,"permalink":"/categories/bushcrafting/","section":"Categories","summary":"","title":"Bushcrafting","type":"categories"},{"content":"","date":"22 April 2024","externalUrl":null,"permalink":"/tags/field-notebook/","section":"Tags","summary":"","title":"Field Notebook","type":"tags"},{"content":"","date":"22 April 2024","externalUrl":null,"permalink":"/tags/hiking/","section":"Tags","summary":"","title":"Hiking","type":"tags"},{"content":"","date":"22 April 2024","externalUrl":null,"permalink":"/tags/logbook/","section":"Tags","summary":"","title":"Logbook","type":"tags"},{"content":"A logbook is a record-keeping tool used in various fields to track information over time. It typically contains entries documenting events, activities, observations, or data related to a particular subject or task. Logbooks can take different forms depending on their purpose, ranging from handwritten notebooks to digital databases.\nHistory of the Logbook Maritime Logbooks: Scientific Logbooks: Aviation Logbooks: How to use a Logbook? Trail Register: Personal Journal: Navigation and Planning: Safety and Emergency Preparedness: Environmental Observations: What information should/could be added to the logbook? Example Logbook - Entry What are the alternatives to Logbooks? What is the difference between a logbook and a Field Notebook or Journal? Logbook: Field Notebook or Journal: An Example Field Notebook Entry. And what have we missed until now? Conclusion In maritime contexts, logbooks record details of a ship\u0026rsquo;s journey, including navigation data, weather conditions, crew activities, and notable events. They serve as crucial legal documents and historical records for ship operations.\nIn scientific research, logbooks record experimental procedures, observations, and results. Researchers use them to maintain an organised work record, essential for documentation, replication of experiments, and publication.\nPilots use logbooks to record flight hours, routes, aircraft types, and other pertinent information for tracking their flying experience and qualifications.\nLogbooks play a vital role in maintaining accurate records, ensuring accountability, and facilitating communication in various fields.\nHistory of the Logbook # The concept of a logbook has ancient roots, dating back to early civilisations, when records were kept on various media, such as stone tablets, papyrus scrolls, and clay tablets. However, the modern form of the logbook has evolved over centuries and across different cultures.\nMaritime Logbooks: # The maritime logbook has a long history, with early examples dating back to ancient seafaring civilisations such as the Phoenicians and Greeks. These early logbooks were likely simple records of voyages, noting landmarks, distances travelled, and basic observations. Logbooks became more standardised and detailed during the Age of Exploration (15th to 17th centuries). Explorers like Christopher Columbus and Ferdinand Magellan kept detailed records of their voyages, including navigational data, weather observations, and encounters with indigenous peoples. By the 18th century, logbooks were standard aboard ships, required by maritime law. They became formalised records, documenting navigational information, crew activities, provisions, and significant events. Logbooks were crucial for navigation, communication with authorities, and legal purposes.\nScientific Logbooks: # In science, they kept logbooks dating back to ancient times, with early scientists like Aristotle and Leonardo da Vinci recording their observations and experiments. The modern scientific logbook began to take shape during the Scientific Revolution (16th to 18th centuries), as the systematic recording of experiments and observations became essential for advancing scientific knowledge. Scientists like Galileo Galilei and Isaac Newton maintained detailed notebooks documenting their experiments, theories, and observations. The tradition of keeping scientific logbooks continues today, with researchers across various disciplines using them to record experimental procedures, data, and findings.\nAviation Logbooks: # The aviation logbook emerged with the development of powered flight in the early 20th century. Pilots began keeping records of their flights, including aircraft type, flight duration, destinations, and maintenance activities. Aviation logbooks are essential for pilots to track their flying experience and qualifications. They are also used by aircraft maintenance personnel to record maintenance and repair activities. Aviation logbooks play a crucial role in ensuring the safety and reliability of aircraft operations, as they provide a detailed history of the aircraft\u0026rsquo;s maintenance and operational status.\nOverall, the history of the logbook is intertwined with the history of human exploration, scientific inquiry, and technological advancement. From ancient seafaring civilisations to modern aviation, logbooks have been indispensable tools for recording, documenting, and communicating information across various domains.\nHow to use a Logbook? # During a hike, a logbook can serve several purposes, depending on the context and preferences of the hiker. Here are some common ways a logbook might be used during a hike:\nTrail Register: # Many popular hiking trails, especially those in remote or wilderness areas, have trail registers or logbooks at trailheads or critical points along the route. Hikers can sign in and provide information such as their name, group size, planned route, and date/time of departure. This safety measure allows authorities to track hikers in case of emergencies or missing persons.\nPersonal Journal: # Some hikers use a logbook as a personal journal to document their hiking experiences. During the hike, they may write about the trail conditions, wildlife sightings, exciting landmarks, and their thoughts and feelings. Keeping a journal can enhance the hiking experience by providing a way to reflect on and remember the adventure.\nNavigation and Planning: # Hikers may use a logbook to record important navigation information, such as trail junctions, distances between landmarks, and waypoints. This can be particularly useful for planning future hikes or sharing route details with other hikers. Recording navigation cues and observations can help hikers stay on track and avoid getting lost.\nSafety and Emergency Preparedness: # In addition to signing trail registers, hikers can use a logbook to document their itinerary and emergency contact information. This information can be crucial for search and rescue teams in an emergency or if the hiker fails to return as planned. Hikers can also use the logbook to record any incidents or injuries encountered during the hike.\nEnvironmental Observations: # Hikers interested in environmental conservation and natural history may use a logbook to record observations of plant and animal species encountered along the trail. They can note the species, location, behaviour, and other relevant details. These observations can contribute to citizen science projects or personal research efforts to understand and protect natural ecosystems.\nA logbook can be a versatile and valuable tool for hikers. It serves practical, personal, and safety-related purposes during a hike. Whether signing a trail register, keeping a journal, recording navigation information, or documenting environmental observations, a logbook can enhance the hiking experience and contribute to a deeper connection with nature.\nWhat information should/could be added to the logbook? # The information added to a logbook during a hike can vary depending on the hike\u0026rsquo;s purpose, the hiker\u0026rsquo;s preferences, and any specific requirements or guidelines for the trail or location. However, here are some common types of information that hikers may choose to include in their logbooks:\nDate and Time : Record the date and time of the hike, including the start and end times if applicable. This helps to establish a timeline of the hike.\nCoordinates : The coordinates from when the entry was written in the field.\nTrailhead or Starting Point : Note the name or location of the trailhead where the hike begins.\nTrail Route : Describe the planned route of the hike, including any specific trails, junctions, or landmarks along the way.\nWeather Conditions : Document the weather conditions at the start of the hike and any changes observed during the hike, such as temperature, precipitation, wind speed, and cloud cover.\nTrail Conditions : Note the condition of the trail, including terrain, obstacles, signage, and any maintenance or safety issues encountered.\nHiking Partners : Record the names of any companions or group members participating in the hike.\nEquipment and Gear : List the gear and equipment carried during the hike, including backpacks, clothing, footwear, navigation tools, safety gear, and any specialised equipment for the terrain or conditions.\nWater and Food : Document the amount of water and Food carried and any sources of water encountered along the trail for refilling supplies.\nWildlife Sightings : Record observations of wildlife encountered during the hike, including species, behaviour, and location.\nScenic Points or Landmarks : Note any noteworthy points of interest or scenic views along the trail.\nNavigation Cues : Document navigation cues such as trail markers, junctions, distances between landmarks, and waypoints for reference during the hike.\nPersonal Reflections : Include personal reflections, thoughts, and feelings about the hike, including highlights, challenges, and memorable moments.\nSafety and Emergency Information : Write down emergency contact information, medical conditions, and any safety precautions taken during the hike.\nConclusion : Summarise the hike\u0026rsquo;s completion, including the total distance covered, any deviations from the planned route, and overall impressions of the experience.\nBy including this information in a logbook, hikers can create a comprehensive record of their hiking adventures, document important details for safety and planning purposes, and preserve memories of their outdoor experiences.\nExample Logbook - Entry # Here\u0026rsquo;s an example of a logbook entry for a day hike:\n-\u0026ndash;\nDate : April 18, 2024 Time : 9:00 AM - 3:00 PM\nTrailhead : Pine Ridge Trailhead, Yosemite National Park\nWeather Conditions : Sunny with a few clouds, temperature around 70°F (21°C), light breeze.\nTrail Route : Took Pine Ridge Trail to Dewey Point, then returned via the same route.\nHiking Partners : Solo hike.\nEquipment and Gear : Backpack with a hydration bladder, hiking boots, trekking poles, sun hat, sunglasses, sunscreen, trail map, GPS device, snacks (granola bars, trail mix), and first aid kit.\nWater and Food : I carried 2 litres of water and snacks for the day, which I refilled at the trailhead.\nTrail Conditions : The trail is in good condition, well-marked, and has occasional patches of mud. There are some fallen branches, but they are easy to navigate around.\nWildlife Sightings : I saw a family of deer near the trailhead and spotted several birds along the trail, including Stellar\u0026rsquo;s jays and woodpeckers.\nScenic Points or Landmarks : Spectacular views of Yosemite Valley and Half Dome from Dewey Point lookout. Took some time to rest and enjoy the scenery.\nNavigation Cues : I followed trail markers and used GPS devices to confirm my location at junctions. The trail signs were clear and easy to follow.\nPersonal Reflections : I enjoyed the solitude of the solo hike and the stunning views from Dewey Point. I felt a sense of accomplishment reaching the lookout and taking in the beauty of Yosemite\u0026rsquo;s wilderness. I am grateful for the opportunity to explore such a magnificent place.\nSafety and Emergency Information : I carried a fully charged cell phone and emergency whistle. I notified a friend of the hiking plans and expected return time. There were no incidents or injuries during the hike.\nConclusion : I completed approximately 8 miles round trip. Overall, it was a rewarding and memorable hike in Yosemite National Park. I am heading back to camp feeling rejuvenated and inspired by nature.\n-\u0026ndash;\nThis logbook entry provides a detailed hike account, including essential information such as trail conditions, wildlife sightings, safety precautions, and personal reflections on the experience.\nWhat are the alternatives to Logbooks? # There are several alternatives to traditional logbooks, depending on the specific needs and preferences of the user. Here are some standard options:\nDigital Logs or Apps : Many hikers, pilots, sailors, and researchers use digital tools such as smartphone apps or computer software to maintain logs. These digital logs can offer various features, such as GPS tracking, photo integration, cloud storage, and data analysis capabilities. Examples include hiking apps like AllTrails, aviation logbook apps like ForeFlight, and scientific data logging software like LabChart.\nField Notebooks or Journals : Instead of a formal logbook, some individuals use field notebooks or journals to record their observations, thoughts, and experiences. These can be simple notebooks or specialised journals for outdoor activities, scientific research, or personal reflection. Field notebooks offer flexibility and creativity in recording information, with sketching, writing, or annotating options.\nOnline Platforms : Online platforms and websites provide alternatives for collaborative logging and data sharing. For example, hikers can use websites like Trailpost or Backpacker\u0026rsquo;s Review to log their hikes and share trail information with other outdoor enthusiasts. Similarly, scientists can use online databases and repositories to store and share research data with colleagues and the broader scientific community.\nCustomised Templates or Forms : Some users create customised templates or forms tailored to their logging needs. These templates can be designed using word processing software, spreadsheet programs, or specialised logbook templates available online. Customised templates allow users to organise information according to their preferences and requirements.\nVoice Recording or Dictation : For hands-free logging, some users opt for voice recording or dictation tools to capture their thoughts and observations during activities such as hiking, flying, or conducting research. Voice recording apps or built-in voice assistants can be convenient alternatives to traditional written logs, especially when manual recording may be impractical.\nWearable Technology : Advancements in wearable technology, such as smartwatches and fitness trackers, offer another alternative for logging activities and collecting data. These devices can track various metrics such as steps taken, distance travelled, heart rate, and elevation gain, providing users with real-time feedback and historical data for analysis.\nOverall, the choice of alternative to logbooks depends on factors such as the nature of the activity, desired features, technology accessibility, and personal preferences for recording and organising information.\nWhat is the difference between a logbook and a Field Notebook or Journal? # The primary difference between a logbook and a field notebook or journal lies in their intended use and format:\nLogbook: # A logbook is a formal record-keeping tool that documents specific information systematically over time. Logbooks typically follow a structured format with predefined sections for recording particular data types or observations. They are often used in professional or regulatory contexts where accuracy, consistency, and accountability are essential, such as maritime operations, aviation, scientific research, and engineering. Logbooks may be required by regulations, standards, or organisational procedures to maintain accurate records for legal, safety, or documentation purposes. Logbook entries are often concise, factual, and standardised to facilitate communication, analysis, and compliance.\nField Notebook or Journal: # A field notebook or journal is a more flexible and informal tool to record observations, thoughts, and experiences during fieldwork, outdoor activities, or personal reflection. Field notebooks typically have blank or lightly ruled pages, allowing users to write, sketch, or annotate as needed. They are commonly used by scientists, researchers, naturalists, artists, writers, and outdoor enthusiasts to document their observations, discoveries, and creative ideas. Field notebooks provide space for capturing detailed descriptions, drawings, diagrams, and personal reflections, fostering creativity and exploration. Unlike logbooks, field notebooks are not usually subject to strict guidelines or regulations, and entries can vary widely in format, content, and style based on the user\u0026rsquo;s preferences and objectives.\nIn summary, while both logbooks and field notebooks serve as tools for recording information, logbooks are typically formal, structured, and standardised for specific purposes such as compliance, documentation, and communication. In contrast, field notebooks are more informal, flexible, and personal, allowing users to capture a wide range of observations and experiences in their style.\nAn Example Field Notebook Entry. # Here\u0026rsquo;s an example of an entry in a field notebook documenting observations during a nature hike:\n-\u0026ndash;\nDate : April 18, 2024\nLocation : Redwood Trail, Muir Woods National Monument\nWeather : Sunny with a light breeze, temperature around 65°F (18°C)\nTime : 10:00 AM - 1:00 PM\nObservations :\nFlora :\nTowering redwood trees (Sequoia sempervirens) dominate the forest canopy, reaching heights of over 250 feet. I noted several specimens with distinctive burls and knotholes. Understory vegetation includes ferns (Polystichum munitum) and sword ferns (Polystichum munitum), with patches of delicate maidenhair ferns (Adiantum spp.) lining the shaded trails. Wildflowers in bloom, including trilliums (Trillium spp.), Pacific bleeding hearts (Dicentra formosa), and western azaleas (Rhododendron occidentale), adding splashes of colour to the forest floor.\nFauna :\nI heard the melodious songs of varied thrushes (Ixoreus naevius) echoing through the forest and the drumming of pileated woodpeckers (Dryocopus pileatus) in the distance. I spotted a western grey squirrel (Sciurus griseus) foraging near a fallen log, its bushy tail twitching as it darted among the undergrowth. Several banana slugs (Ariolimax spp.) were observed on the damp forest floor, their bright yellow bodies contrasting against the rich brown leaf litter.\nGeology and Terrain :\nI noted the presence of serpentine rock outcrops along the trail, characterised by their greenish hue and sparse vegetation. Serpentine soils support unique plant communities adapted to their nutrient-poor and drought-prone conditions. Trail terrain varied from gentle slopes to steep inclines, with occasional rocky sections and exposed roots. Trail maintenance is evident, with well-maintained paths and sturdy bridges crossing seasonal streams.\nCultural and Historical Significance :\nReflecting on Muir Woods\u0026rsquo; historical significance as a protected old-growth redwood forest and its role in inspiring conservation efforts, I appreciated the tranquillity and sense of awe instilled by the ancient trees and pristine surroundings.\nSketches and Notes :\nSketch of a topographic map that shows an essential point in the field\nSketch of a towering redwood tree with annotations noting its size and distinctive features\nSketch of a banana slug with descriptive notes on its anatomy and habitat\nReflections :\nImmersed in the beauty and serenity of Muir Woods, I feel grateful for the opportunity to connect with nature and explore this ancient forest. I am inspired to learn more about the ecology and conservation of redwood ecosystems.\n-\u0026ndash;\nThis entry in the field notebook provides the following:\nA detailed account of observations made during a Muir Woods National Monument hike. Capturing botanical, zoological, geological, and cultural aspects of the environment while incorporating sketches. Notes. Personal reflections. And what have we missed until now? # The importance of including coordinates in each logbook entry varies depending on the activity\u0026rsquo;s context and the logbook\u0026rsquo;s specific goals. Here are some factors to consider regarding the significance of coordinates:\nNavigation and Wayfinding : Coordinates can be essential for activities such as hiking, mountaineering, boating, or flying. Including coordinates in logbook entries allows hikers, pilots, sailors, or outdoor enthusiasts to accurately pinpoint their location on maps and charts, helping them stay on course and navigate to desired destinations.\nSafety and Emergency Response : Coordinates are crucial in safety and emergency response situations. If a hiker becomes lost or injured, having accurate coordinates recorded in a logbook can expedite search and rescue efforts by providing rescuers with precise location information. Similarly, distressed pilots or sailors can transmit their coordinates to emergency services for assistance.\nDocumentation and Verification : Coordinates serve as objective, verifiable data points that can validate the accuracy and credibility of logbook entries. Including coordinates adds precision and detail to observations, measurements, or discoveries documented in the logbook, enhancing the reliability and utility of the information for research, analysis, or documentation purposes.\nMapping and Analysis : Coordinates enable the integration of logbook data with mapping software, geographic information systems (GIS), or spatial analysis tools. By georeferencing logbook entries, users can visualise and analyse spatial patterns, distributions, and relationships, leading to insights and discoveries that may not be apparent from textual descriptions alone.\nContext and Use Case : The importance of coordinates in logbook entries ultimately depends on the logbook\u0026rsquo;s specific context and use case. In some scenarios, such as scientific research, environmental monitoring, or geocaching, precise location data may be critical for achieving objectives and answering research questions. In other cases, such as personal reflection or creative expression, coordinates may be less relevant and can be omitted if they do not contribute to the logbook\u0026rsquo;s intended purpose.\nIn summary, while coordinates can be highly valuable and even essential in specific contexts, their importance in each logbook entry should be evaluated based on factors such as the nature of the activity, safety considerations, data requirements, and the intended use of the logbook.\nConclusion # Using a logbook or a similar tool for documenting activities, observations, and experiences offers numerous benefits across various contexts. Here\u0026rsquo;s a conclusion summarising the advantages:\nOrganisation and Documentation : Logbooks provide a structured format for organising information, ensuring that important details are recorded systematically. Whether it\u0026rsquo;s recording navigational data during a flight, documenting scientific observations in the field, or logging personal reflections during a hike, a logbook helps maintain a comprehensive record of activities over time.\nAccuracy and Accountability : Logbooks promote accuracy and accountability by requiring users to record information in real-time or shortly after an event. This is especially crucial in aviation, maritime operations, scientific research, and regulatory compliance, where precise documentation is essential for safety, legal, or regulatory purposes.\nCommunication and Collaboration : Logbooks facilitate communication and collaboration among team members, colleagues, or stakeholders by providing a shared repository of information. Logbooks are a platform for exchanging information and insights, whether it\u0026rsquo;s sharing navigational data with air traffic control, collaborating on a research project, or exchanging trail conditions with fellow hikers.\nDecision Making and Analysis : Logbooks provide valuable data for decision-making, analysis, and problem-solving. By documenting observations, trends, and incidents over time, logbooks enable users to identify patterns, assess performance, and make informed decisions based on historical data. This is particularly important in fields where data-driven decisions are critical, such as emergency response, scientific research, and project management.\nReflection and Learning : Beyond practical utility, logbooks offer a space for personal reflection, learning, and growth. Whether it\u0026rsquo;s reflecting on lessons learned from a challenging hike, analysing experimental results in a laboratory setting, or documenting personal achievements and milestones, logbooks provide a platform for self-expression and introspection.\nIn conclusion, using a logbook or a similar tool enhances organisation, accuracy, communication, decision-making, and personal reflection across various activities and disciplines. Whether for professional, recreational, or personal purposes, the disciplined practice of logging activities and observations fosters accountability.\n","date":"22 April 2024","externalUrl":null,"permalink":"/posts/what-is-a-logbook/","section":"Posts","summary":"A logbook is a record-keeping tool used in various fields to track information over time. It typically contains entries documenting events, activities, observations, or data related to a particular subject or task. Logbooks can take different forms depending on their purpose, ranging from handwritten notebooks to digital databases.\n","title":"What is a Logbook?","type":"posts"},{"content":"Discover the hidden dangers of bidirectional control characters! We dive deep into how these essential text-rendering tools can be exploited to manipulate digital environments. Learn about their security risks, from filename spoofing to deceptive URLs, and uncover the crucial strategies to safeguard against these subtle yet potent threats. Understand how to protect your systems in a multilingual world. Join to ensure your digital security is not left to chance!\nKey Bidirectional Control Characters Left-to-Right Mark (LRM) - U+200E: Right-to-Left Mark (RLM) - U+200F: Left-to-Right Embedding (LRE) - U+202A: Right-to-Left Embedding (RLE) - U+202B: Pop Directional Formatting (PDF) - U+202C: Left-to-Right Override (LRO) - U+202D: Right-to-Left Override (RLO) - U+202E: Uses and Applications Some Demos Java Demo - Right-to-Left Override (RLO) Attack Explanation Java Demo - Right-to-Left Mark (RLM) Attack Explanation But why is this a security issue? File Name Spoofing Phishing Attacks Code Obfuscation Misleading Data and Database Entries User Interface Deception Addressing the Security Risks Input Validation and Sanitization Secure Default Configurations User and Administrator Education Enhanced Monitoring and Logging Security Policies and Procedures Technological Solutions Conclusion: Key Bidirectional Control Characters # Bidirectional control characters (often abbreviated as bidi control characters) are special characters used in text encoding to manage the direction of text flow. This is crucial for languages read right-to-left (RTL), like Arabic and Hebrew, when mixed with left-to-right (LTR) languages like English. These characters help to ensure that the text is displayed in the correct order, regardless of the directionality of its parts.\nHere are some of the common bidi control characters defined in the Unicode standard:\nLeft-to-Right Mark (LRM) - U+200E: # They are used to set the direction of the text from left to right. It is particularly useful when embedding a small piece of LTR text within a larger segment of RTL text.\nRight-to-Left Mark (RLM) - U+200F: # This setting sets the direction of the text to right-to-left. It is used when embedding a small RTL text within a larger segment of LTR text.\nLeft-to-Right Embedding (LRE) - U+202A: # They are used to start a segment of LTR text within an RTL environment. This embedding level pushes onto the directional status stack.\nRight-to-Left Embedding (RLE) - U+202B: # They are used to start a segment of RTL text within an LTR environment.\nPop Directional Formatting (PDF) - U+202C: # They are used to end a segment of embedded text, popping the last direction from the stack and returning to the previous directional context.\nLeft-to-Right Override (LRO) - U+202D: # It forces the text within its scope to be treated as left-to-right text, regardless of its directionality. This is useful for reordering sequences of characters.\nRight-to-Left Override (RLO) - U+202E: # Forces the text within its scope to be treated as right-to-left text, even if it is typically LTR. This can be used to display text backwards, which might be used for effect or in specific contexts.\nUses and Applications # Bidirectional control characters are essential for the following:\nMultilingual Documents : Ensuring coherent text flow when documents contain multiple languages with different reading directions.\nUser Interfaces : Proper text rendering in software that supports multiple languages.\nData Files : Manage data display in multiple languages with different directionalities.\nSome Demos # Bidirectional control characters can pose security risks. They can be used to obscure the true intent of a code or text, leading to what is known as a \u0026ldquo;bidirectional text attack.\u0026rdquo; For instance, filenames could appear to end with a harmless extension like \u0026ldquo;.txt\u0026rdquo; when they end with a dangerous one like \u0026ldquo;.exe\u0026rdquo; reversed by bidi characters. As a result, users might need to be more informed about the nature of the files they interact with.\nSecurity-aware text editors and systems often have measures to detect and appropriately display or alert users about the presence of bidirectional control characters to mitigate potential security risks.\nHere\u0026rsquo;s a simple Java demo that illustrates how bidirectional control characters can be used to create misleading filenames. This can demonstrate the potential danger, particularly in environments where filenames are manipulated or displayed based on user input.\nJava Demo - Right-to-Left Override (RLO) Attack # This demo will:\nCreate a seemingly harmless text file named \u0026ldquo;txt.exe\u0026rdquo; using bidirectional control characters. The file will output the actual and displayed names to show the discrepancy.\nimport java.io.File; import java.io.IOException; public class BidiDemo { public static void main(String[] args) { // U+202E is the Right-to-Left Override (RLO) character String normalName = \u0026#34;report.txt\u0026#34;; String deceptiveName = \u0026#34;report\u0026#34; + \u0026#34;\\u202E\u0026#34; + \u0026#34;exe.txt\u0026#34;; // Try to create files with these names createFile(normalName); createFile(deceptiveName); // Print what the names look like to the Java program System.out.println(\u0026#34;Expected file name: \u0026#34; + normalName); System.out.println(\u0026#34;Deceptive file name appears as: \u0026#34; + deceptiveName); } private static void createFile(String fileName) { File file = new File(fileName); try { if (file.createNewFile()) { System.out.println(\u0026#34;File created: \u0026#34; + file.getName()); } else { System.out.println(\u0026#34;File already exists: \u0026#34; + file.getName()); } } catch (IOException e) { System.out.println(\u0026#34;An error occurred while creating the file: \u0026#34; + fileName); e.printStackTrace(); } } } Explanation # Creation of Names : The deceptive file name is created using the right-to-left override character (U+202E). This causes the part of the filename after the bidi character to be interpreted as right-to-left, making \u0026ldquo;exe.txt\u0026rdquo; look like \u0026ldquo;txt.exe\u0026rdquo; in some file systems and interfaces.\nFile Creation : The program attempts to create files with standard and deceptive names.\nOutput Differences : When printed, the deceptive name will show the filename reversed after the bidi character, potentially misleading users about the file type and intent.\nTo see the effect:\n- Compile and run the Java program.\n- Check the output and the file system to observe how the filenames are displayed.\nJava Demo - Right-to-Left Mark (RLM) Attack # Let\u0026rsquo;s examine a Java example that demonstrates how a Right-to-Left Mark (RLM) can be critical in ensuring the correct display and handling of mixed-direction text. This example will simulate a simple scenario where Arabic and English text are combined, highlighting how the RLM character helps maintain the intended order of words.\nThis Java example will:\n1. Combine English and Arabic text in a single string.\n2. Use the Right-to-Left Mark (RLM) to manage the display order correctly.\n3. Print out the results to illustrate the effect of using RLM.\npublic class RLMExample { public static void main(String[] args) { // Arabic reads right to left, English left to right String englishText = \u0026#34;Version 1.0\u0026#34;; String arabicText = \u0026#34;الإصدار\u0026#34;; // Concatenate without RLM String withoutRLM = arabicText + \u0026#34; \u0026#34; + englishText; // Concatenate with RLM String withRLM = arabicText + \u0026#34;\\u200F\u0026#34; + \u0026#34; \u0026#34; + englishText; // Print the results System.out.println(\u0026#34;Without RLM: \u0026#34; + withoutRLM); System.out.println(\u0026#34;With RLM: \u0026#34; + withRLM); } } Explanation # Arabic and English Text : Arabic is inherently right-to-left, whereas English is left-to-right.\nConcatenation without RLM : Depending on the environment, simply concatenating Arabic and English text might not always display correctly, as the directionality of the English text can disrupt the flow of the Arabic.\nConcatenation with RLM : By inserting a Right-to-Left Mark after the Arabic text but before the English text, the English part is correctly treated as part of the right-to-left sequence. This ensures the English text is read in its natural order but positioned correctly within the overall RTL context.\nWhen you run this program, especially in a console or environment that supports bidirectional text:\nThe \u0026ldquo;Without RLM\u0026rdquo; output may show the English text misplaced or improperly aligned relative to the Arabic text.\nThe \u0026ldquo;With RLM\u0026rdquo; output should show the English text correctly placed and maintain the natural reading order of both languages.\nThis example underscores the importance of RLM in software and user interfaces dealing with multilingual data. It ensures that text is presented in a way that respects the reading order of different languages. Proper handling of bidirectional text is crucial in applications ranging from document editors to web content management systems.\nBut why is this a security issue? # Bidirectional control characters like the Right-to-Left Mark (RLM) are a security concern primarily due to their ability to obscure the true intent of text and data. This ability can be exploited in various ways to mislead users or automated systems about the content or function of data, leading to potential security vulnerabilities. Here are some specific scenarios where this becomes critical:\nFile Name Spoofing # One of the most common security issues related to bidirectional control characters is file name spoofing. Attackers can use bidi characters to reverse the order of characters in a file\u0026rsquo;s extension in file names, making a malicious executable file appear as a harmless type, such as a text file. For instance, the file named doc.exe might be displayed as exe.cod in systems that do not handle bidi characters properly, tricking users into thinking it\u0026rsquo;s merely a document.\nPhishing Attacks # In phishing emails or misleading links, bidi characters can be used to reverse parts of a URL to mimic a trusted domain, leading users to malicious sites. For example, what appears to be example.com in reversed parts could be a link to an entirely different and dangerous site, exploiting the user\u0026rsquo;s trust in familiar-looking URLs.\nCode Obfuscation # Developers or malicious coders might use bidi characters to obscure code logic or comments in software, making it difficult for security analysts or automated tools to assess the code\u0026rsquo;s behaviour accurately. This can hide malicious functions or bypass security audits.\nMisleading Data and Database Entries # Bidi characters can be used to reverse strings in database entries, potentially leading to incorrect or misleading data processing. This could be exploited to bypass filters and validation checks or to intentionally corrupt data integrity.\nUser Interface Deception # In applications with user interfaces that display user input data, bidi characters can create a misleading representation of that data. This could need to be clarified for users or lead them to make incorrect decisions based on incorrectly displayed information.\nAddressing the Security Risks # Addressing the security risks associated with bidirectional control characters (bidi characters) requires a multifaceted approach that includes technical safeguards and user education. Here are more detailed strategies that organizations and software developers can employ to mitigate these risks:\nInput Validation and Sanitization # Strict Validation Rules : Implement strict validation rules that check for the presence of bidi characters in sensitive contexts such as file names, URLs, and input forms. This validation should identify and flag or reject unexpected or unauthorized use of these characters.\nCharacter Filtering : For applications not requiring bidi characters, remove them from inputs during the data entry or ingestion process. For applications where such characters are necessary, ensure they are used correctly and safely.\nEncoding Techniques : Use encoding techniques to handle potentially dangerous characters safely. For example, HTML entities can encode bidi characters in web applications, preventing them from being processed as active components of the code.\nSecure Default Configurations # Display Controls : Configure systems and applications to visually distinguish or neutralize bidi characters, particularly in environments where their use is rare or unexpected. This could involve displaying their unicode point instead of the character or providing visual indicators of text direction changes.\nLimit Usage Contexts : Restrict the contexts in which bidi characters can be used, especially in identifiers like usernames, filenames, and URLs, unless there is a specific need for them.\nUser and Administrator Education # Awareness Training : Conduct regular training sessions for users and administrators about potentially misusing bidi characters and other Unicode anomalies. Include real-world examples of how these features can be exploited.\nBest Practices for Content Creation : Educate content creators on the correct and safe use of bidi characters, emphasizing the security aspects of text directionality in content that will be widely distributed or used in sensitive environments.\nEnhanced Monitoring and Logging # Anomaly Detection : Use advanced monitoring tools to detect unusual bidi character usage patterns in system logs, network traffic, or transaction data. This can help identify potential attacks or breaches early.\nAudit Trails : Maintain robust audit trails, including detailed logging of input validation failures and other security-related events. This can help with forensic analysis and understanding attack vectors after a security incident.\nSecurity Policies and Procedures # Clear Policies : Develop and enforce clear security policies regarding handling bidi characters. This includes guidelines for developers handling text input and output and policies for content managers reviewing and approving content.\nIncident Response : Include the misuse of bidi characters as a potential vector in your organization\u0026rsquo;s incident response plan. Prepare specific procedures to respond to incidents involving deceptive text or file manipulations.\nTechnological Solutions # Development Frameworks and Libraries : Utilize frameworks and libraries that inherently handle bidi characters safely and transparently. Ensure that these tools are up-to-date and configured correctly.\nUser Interface Design : Design user interfaces that inherently mitigate the risks posed by bidi characters, such as displaying full file extensions and using text elements that visually separate user input from system text.\nImplementing these strategies requires a coordinated effort between software developers, security professionals, system administrators, and end-users. Organizations can significantly reduce the risks of bidi characters and other related security threats by adopting comprehensive measures.\nConclusion: # In conclusion, while often overlooked, the security risks associated with bidirectional control characters are significant and can have profound implications for individuals and organizations. These characters can be exploited in various deceptive ways, from file name spoofing and phishing attacks to code obfuscation and misleading data presentations. To effectively mitigate these risks, a comprehensive and multi-layered approach is necessary.\nThis approach should include stringent input validation and sanitization processes to filter out or safely handle bidi characters where they are not needed and to ensure they are used appropriately where they are necessary. Secure default configurations that visually indicate the presence and effect of bidi characters can help prevent their misuse, while robust monitoring and logging can aid in detecting and responding to potential security threats.\nEducation also plays a crucial role. Users and administrators need to be aware of how bidi characters can be used maliciously, and developers need to be informed about best practices for handling such characters in their code. Security policies must be clear and enforced, with specific guidelines on handling bidi characters effectively and safely.\nFinally, employing technological solutions that can handle these characters appropriately and designing user interfaces that mitigate their risks will further strengthen an organization\u0026rsquo;s defence against the security vulnerabilities introduced by bidirectional control characters. By addressing these issues proactively, we can safeguard the integrity of digital environments and protect sensitive information from being compromised.\n","date":"19 April 2024","externalUrl":null,"permalink":"/posts/the-hidden-dangers-of-bidirectional-characters/","section":"Posts","summary":"Discover the hidden dangers of bidirectional control characters! We dive deep into how these essential text-rendering tools can be exploited to manipulate digital environments. Learn about their security risks, from filename spoofing to deceptive URLs, and uncover the crucial strategies to safeguard against these subtle yet potent threats. Understand how to protect your systems in a multilingual world. Join to ensure your digital security is not left to chance!\n","title":"The Hidden Dangers of Bidirectional Characters","type":"posts"},{"content":"Audio steganography is a technique for hiding information within an audio file so that only the intended recipient knows of the hidden data\u0026rsquo;s existence. This method belongs to the broader field of steganography, which itself is a subset of security systems and comes from the Greek words \u0026ldquo;steganos,\u0026rdquo; meaning covered, and \u0026ldquo;graphein,\u0026rdquo; meaning writing.\nThe primary objective of audio steganography is to conceal the presence of secret data by embedding it into an audio signal without noticeable degradation of the signal. Unlike cryptography, which secures the contents of a message through encryption, steganography focuses on concealing the fact that there is a hidden message.\nMethods of Audio Steganography # There are various techniques used to embed data within audio files, including:\nLeast Significant Bit (LSB) Insertion: # This is one of the simplest methods where bits of the hidden data are inserted into the least significant bits of the audio file. Because these modifications are slight, they are generally imperceptible to the human ear.\nPhase Coding: # Phase coding is a sophisticated technique used in audio steganography to embed information within an audio file by manipulating the phase of the sound signal. This method leverages the fact that the human auditory system is far less sensitive to phase changes than it is to changes in amplitude or frequency. As such, phase coding can be a very effective way to hide information without noticeable changes to the sound quality as perceived by human listeners.\nHow Phase Coding Works # Phase coding is typically applied to the phase spectrum of a sound signal. The process involves several steps:\nSignal Decomposition: The original audio signal is divided into segments using a Fast Fourier Transform (FFT) transform. This decomposition converts the time-domain signal into the frequency domain, where each component has an amplitude and a phase.\nPhase Manipulation: The phase of the audio file\u0026rsquo;s initial segment (or a reference segment) is adjusted to embed the secret data. The phases of subsequent segments are then altered to ensure a smooth transition between segments, maintaining the perceptual integrity of the audio signal. This is crucial to prevent artefacts that could be detectable by the human ear.\nData Embedding: The binary data to be hidden is typically encoded in the phase changes between successive segments. A simple method might involve using the presence or absence of a phase shift as a binary one or zero respectively.\nSignal Reconstruction: After the phase modification, the segments are transformed back to the time domain, recombining them to produce the final audio signal with the hidden data embedded.\nAdvantages of Phase Coding # Subtlety: Since human ears are not particularly sensitive to phase variations, especially in complex sounds with many overlapping frequencies, this method can hide data effectively without audible distortion.\nRobustness to Compression: Phase coding can be relatively robust against lossy compression, especially compared to methods like LSB insertion, which such processes can disrupt.\nChallenges and Considerations # Complexity: Implementing phase coding requires careful handling to maintain the coherence of the phase between segments. Poor implementation can result in noticeable audio artifacts.\nData Capacity: While phase coding is discreet, it typically offers lower data capacity compared to other steganographic techniques. This is because excessive manipulation of phase information can lead to perceptible sound quality degradation or inconsistencies.\nDependency on Sound Content: The effectiveness of phase coding can depend heavily on the type of audio being used. Complex signals with many frequency components (like music) offer more opportunities for phase manipulation without detection than more straightforward signals (like speech).\nIn summary, phase coding is a powerful but complex technique in audio steganography, offering a high level of discretion for embedding data within audio files. Its successful application requires careful handling to balance data capacity, sound quality, and robustness against signal processing.\nSpread Spectrum: # In this method, the secret message is spread across the frequency spectrum of the audio file. This spreading makes the message less susceptible to intentional or unintentional modifications.\nThis technique is based on spread spectrum communication technology principles, initially developed for military use to ensure secure and robust communication over radio waves.\nHow Spread Spectrum Works # In the context of audio steganography, spread spectrum involves embedding a secret message into an audio signal by slightly altering its frequency components. The process typically follows these steps:\nData Preparation: The data to be hidden is first prepared, often by encoding it in a binary format. This binary data is then spread out over a larger bandwidth than it would normally occupy, which helps in disguising its presence.\nSignal Spreading: The spread data is combined with a pseudo-noise code (PN code) or a similar key that only the sender and intended recipient know. The PN code has properties similar to noise but with a known structure, making it possible to detect and decode the data without being detected by unintended listeners.\nModulation: The combined data and PN code modulate the host audio signal, typically using techniques like Direct Sequence Spread Spectrum (DSSS) or Frequency Hopping Spread Spectrum (FHSS). In DSSS, the signal is spread across a wide range of frequencies based on the PN code. In FHSS, the signal frequency hops rapidly in a pattern defined by the PN code.\nIntegration into Audio Signal: The modulated signal is then subtly integrated into the audio file. This integration is done so that the modifications to the audio are imperceptible to the human ear but can be detected and decoded by analyzing the audio spectrum with the correct key.\nAdvantages of Spread Spectrum # Robustness: Spread spectrum techniques are highly resistant to interference and noise. Because the data is spread across a wide band of frequencies, it remains intact even if parts of the signal are disrupted or lost.\nSecurity: The use of a PN code makes the hidden data difficult to detect and decode without the correct key, providing an additional layer of security.\nLow Detectability: The hidden data\u0026rsquo;s wide distribution across the frequency spectrum and low amplitude make it difficult to detect through casual listening or even with spectral analysis without prior knowledge.\nChallenges and Considerations # Complexity: Implementing spread spectrum in audio steganography requires sophisticated signal processing techniques and careful tuning to ensure that the hidden data does not affect the quality of the audio.\nBandwidth Requirements: The method requires more bandwidth to spread the data, which can be a limitation in environments where bandwidth is constrained.\nDependency on Audio Content: Like other steganographic methods, the effectiveness and capacity of the spread spectrum can depend on the nature of the audio content. Complex audio signals with a wide dynamic range and rich frequency content are better for hiding data.\nEcho Hiding: # This involves introducing an echo into the discrete signal. Parameters like the amplitude, decay rate, and offset of the echo can be manipulated to embed data. This method capitalizes on the auditory characteristics of human perception, specifically how humans perceive echoes, to hide data effectively without noticeably altering the quality of the audio.\nHow Echo Hiding Works # Echo hiding works by manipulating the properties of echo—such as delay, decay, and amplitude—to encode data. The basic steps involved in echo hiding are:\nEcho Creation: Echoes are artificially created and superimposed onto the original audio signal. The parameters of these echoes—such as the delay (time between the original sound and its echo) and the decay rate (how quickly the echo fades away)—are crucial.\nData Embedding: Binary data is encoded into the audio signal by varying the parameters of the echoes. For instance, a short delay might represent a binary \u0026lsquo;0\u0026rsquo; while a longer delay might represent a binary \u0026lsquo;1\u0026rsquo;. The amplitude of the echo can also be used to encode data, with different levels of loudness representing different data bits.\nParameter Control: To remain imperceptible, the echoes must be subtle. The echo delay is typically kept within the range of 1 to 3 milliseconds, as delays shorter than 1 millisecond are generally not perceived as echoes, and those longer than 3 milliseconds can begin to be perceived as discrete repeats rather than reverberation or natural echo.\nSignal Synthesis: After encoding the data, the modified signal (original plus echo) is synthesized back into a coherent audio stream. This new audio stream contains the hidden information encoded within the echo parameters but should sound nearly identical to the original to an unsuspecting listener.\nAdvantages of Echo Hiding # Imperceptibility: Since the echoes are subtle and use natural auditory phenomena, the modifications are typically imperceptible to casual listeners.\nRobustness: Echo properties can be robust against certain types of signal processing, such as compression and transmission over noisy channels because the characteristics of the echo (especially if embedded in a robust part of the audio spectrum) can remain detectable even if the quality of the audio is somewhat degraded.\nCompatibility: Echo hiding does not require significant alteration of the frequency content of the audio signal, which helps preserve the original quality and characteristics of the audio.\nChallenges and Considerations # Detection and Removal: While robust against some forms of manipulation, sophisticated audio analysis tools designed to identify and modify echo characteristics can detect and potentially remove echoes.\nCapacity Limitations: Compared to other steganographic methods, echo hiding generally hides a limited amount of data. Overloading the signal with too many echoes can make it more detectable and degrade the audio quality.\nDependency on Content: The effectiveness of echo hiding can depend on the nature of the audio content. Audio signals with lots of natural variation and existing reverberation may mask the steganographic echoes better than very dry, sparse, or highly dynamic signals.\nFrequency Masking: # Frequency masking is an audio steganography technique that exploits the limitations of human auditory perception to hide information within an audio file. It leverages a phenomenon known as \u0026ldquo;auditory masking,\u0026rdquo; where certain sounds become inaudible in the presence of other, louder sounds at similar frequencies. This technique is particularly subtle because it embeds data in a way that is naturally concealed by the characteristics of the audio itself.\nHow Frequency Masking Works # The basic concept of frequency masking involves embedding hidden data into parts of the audio spectrum where the presence of louder, dominant sounds will mask it. The process typically involves the following steps:\nAnalysis of Audio Spectrum: The first step is to analyze the frequency spectrum of the host audio signal to identify potential masking opportunities. This involves finding frequency ranges where louder sounds are likely to mask quieter ones.\nData Preparation: The data intended for hiding is prepared, usually encoded into a binary format. This data needs to be modulated or otherwise processed to fit within the selected masked frequencies.\nEmbedding Data: The prepared data is then embedded into the quieter parts of the audio spectrum, specifically within the critical bands of frequency where masking is most effective. Critical bands are frequency ranges in which the human ear processes sound as a single auditory event.\nSignal Synthesis: After embedding the data, the audio signal is reconstructed to include the hidden data. This process must be handled delicately to ensure that the modifications do not become perceptible, maintaining the quality and integrity of the original audio.\nAdvantages of Frequency Masking # Imperceptibility: Because the hidden data is placed in regions where louder sounds naturally mask it, it is very difficult to detect without specific, sophisticated analysis tools.\nRobustness to Compression: Frequency masking can be relatively robust against some forms of audio compression, particularly if the embedded data is strategically placed in less compressible parts of the spectrum.\nUtilization of Auditory Phenomena: This method uses a natural auditory phenomenon, making it a very organic form of steganography.\nChallenges and Considerations # Complex Signal Analysis Required: Effective use of frequency masking requires detailed analysis of the audio signal’s spectral properties, which can be complex and computationally intensive.\nLimited Data Capacity: The amount of data that can be hidden is generally limited to the available masked regions, which may not be extensive depending on the audio content.\nDependency on Audio Content: The success of frequency masking heavily depends on the nature of the audio file. Audio tracks with dense and varied spectral content provide more opportunities for masking than simpler, cleaner tracks.\nApplications # Audio steganography has various applications across multiple fields. Some of these include:\nSecure Communications: These are used by organizations and individuals to communicate sensitive information discreetly.\nWatermarking: Audio files can be watermarked to assert ownership, much like watermarking images or videos.\nCovert Operations: Used by law enforcement and military for operations requiring secure and stealthy communication methods.\nChallenges # Despite its advantages, audio steganography faces several challenges:\nRobustness: # The steganographic information must remain intact even if the audio file undergoes compression, format conversion, or other types of digital processing.\nImperceptibility: # The alterations made to embed the data should not be detectable by normal hearing, as this would compromise the steganographic integrity.\nCapacity: # The amount of data that can be hidden is generally limited by the size of the host file and the technique used, which could be restrictive for larger data needs.\nConclusion # Audio steganography offers a unique way to conceal information within audio files, making it a valuable tool for security and privacy in digital communications. Its effectiveness lies in its ability to hide information in plain sight, providing an added layer of security through obscurity. However, its success and reliability depend heavily on the choice of technique and the nature of the application. As technology evolves, so do steganography methods, which are continuously improving to meet the demands of modern security needs.\n","date":"17 April 2024","externalUrl":null,"permalink":"/posts/audio-steganography-in-more-detail/","section":"Posts","summary":"Audio steganography is a technique for hiding information within an audio file so that only the intended recipient knows of the hidden data’s existence. This method belongs to the broader field of steganography, which itself is a subset of security systems and comes from the Greek words “steganos,” meaning covered, and “graphein,” meaning writing.\n","title":"Audio Steganography In More Detail","type":"posts"},{"content":"","date":"17 April 2024","externalUrl":null,"permalink":"/tags/steganography/","section":"Tags","summary":"","title":"Steganography","type":"tags"},{"content":"Steganography is the practice of concealing a message, file, image, or video within another message, file, image, or video. Unlike cryptography, which focuses on making a message unreadable to unauthorised parties, steganography aims to hide the message\u0026rsquo;s existence. The word \u0026ldquo;steganography \u0026quot; is derived from the Greek words \u0026ldquo;steganos ,\u0026rdquo; meaning \u0026ldquo;covered ,\u0026rdquo; and \u0026ldquo;graphein ,\u0026rdquo; meaning \u0026ldquo;to write.\u0026rdquo;\nAn example from history Example of using Steganography in Cybersecurity How to do it practically in Java Text Steganography: Image Steganography: Audio Steganography: Video Steganography: File Steganography: Conclusion: An example from history # One notable historical example of steganography involves using invisible ink during the American Revolutionary War. The British and American forces employed various forms of steganography to conceal messages and strategic information.\nIn particular, the Culper Spy Ring, a clandestine network of American spies operating during the Revolutionary War, extensively used invisible ink to communicate covertly. One member of the ring, invisible ink expert James Jay, developed a secret method for creating invisible ink using simple household ingredients such as lemon juice or milk.\nThe Culper Spy Ring would write messages with this invisible ink between the lines of innocent-looking letters or on the blank spaces of documents. To reveal the hidden messages, the recipient would apply heat or special chemicals, causing the invisible ink to darken and become visible.\nThis use of steganography allowed the Culper Spy Ring to transmit crucial intelligence about British troop movements, plans, and other sensitive information without detection by British authorities. The effective use of invisible ink by the Culper Spy Ring played a significant role in aiding the American cause during the Revolutionary War and exemplifies the historical importance of steganography in espionage and covert operations.\nhttps://youtu.be/tYeC3A57wgc\nExample of using Steganography in Cybersecurity # A modern example of steganography in cybersecurity involves concealing malicious code within seemingly innocuous digital files to bypass security measures and deliver malware to targeted systems. Cybercriminals often employ this technique to evade detection by traditional antivirus software and intrusion detection systems.\nFor instance, attackers may embed malicious payloads, such as Trojans or ransomware, within images, audio files, or documents using steganography techniques. The carrier files appear unchanged to the naked eye or standard file analysis tools, making it difficult for security solutions to detect hidden malware.\nOnce the steganographically encoded file reaches the target system, the attacker can extract and execute the concealed payload, thereby compromising the system and initiating malicious activities.\nTo combat this threat, cybersecurity professionals utilise advanced threat detection technologies capable of analysing files for signs of steganographic manipulation. These tools employ various techniques, such as statistical analysis, anomaly detection, and signature-based detection, to identify suspicious patterns or deviations from expected file structures.\nAdditionally, cybersecurity awareness training programs educate users about the risks associated with opening files from untrusted sources and emphasise the importance of maintaining up-to-date security software to mitigate the threat of steganography-based attacks.\nAll Code Examples are on GitHub: https://github.com/svenruppert/Steganography\nHow to do it practically in Java # Steganography techniques can involve various methods, such as:\nThe source code you will find on GitHub under the following URL:\nhttps://github.com/svenruppert/Steganography\nText Steganography: # Embedding secret messages within text documents by altering spacing, font styles, or using invisible characters.\nHere\u0026rsquo;s a simple example of text steganography in Java using a basic technique called whitespace steganography. In this example, we\u0026rsquo;ll hide a secret message within a text document by manipulating the whitespace between words.\npublic class TextSteganographyExample { 4 5 public static void main(String[] args) { 6 String binaryData = \u0026#34;101\u0026#34;; // Binary data to hide 7 String sourceText = \u0026#34;Hello\\nWorld\\nThis is a test\\n\u0026#34;; // Source text to hide data in 8 String hiddenText = hideData(sourceText, binaryData); 9 10 System.out.println(\u0026#34;Text without hidden data:\u0026#34;); 11 System.out.println(sourceText); 12 System.out.println(\u0026#34;Text with hidden data:\u0026#34;); 13 System.out.println(hiddenText); 14 15 //extract the hidden message 16 String extractedData = extractData(hiddenText); 17 System.out.println(\u0026#34;extractedData = \u0026#34; + extractedData); 18 } 19 20 public static String hideData(String sourceText, String binaryData) { 21 Scanner scanner = new Scanner(sourceText); 22 StringBuilder hiddenTextBuilder = new StringBuilder(); 23 int index = 0; 24 while (scanner.hasNextLine() \u0026amp;\u0026amp; index \u0026lt; binaryData.length()) { 25 String line = scanner.nextLine(); 26 // Append a space for \u0026#39;0\u0026#39;, or a tab for \u0026#39;1\u0026#39; 27 char appendChar = binaryData.charAt(index) == \u0026#39;0\u0026#39; ? \u0026#39; \u0026#39; : \u0026#39;\\t\u0026#39;; 28 hiddenTextBuilder.append(line).append(appendChar).append(\u0026#34;\\n\u0026#34;); 29 index++; 30 } 31 // If there\u0026#39;s more of the source text, add it as is 32 while (scanner.hasNextLine()) { 33 hiddenTextBuilder.append(scanner.nextLine()).append(\u0026#34;\\n\u0026#34;); 34 } 35 scanner.close(); 36 return hiddenTextBuilder.toString(); 37 } 38 39 public static String extractData(String hiddenText) { 40 Scanner scanner = new Scanner(hiddenText); 41 StringBuilder binaryDataBuilder = new StringBuilder(); 42 while (scanner.hasNextLine()) { 43 String line = scanner.nextLine(); 44 if (line.endsWith(\u0026#34;\\t\u0026#34;)) { 45 // If line ends with a tab, append \u0026#39;1\u0026#39; 46 binaryDataBuilder.append(\u0026#39;1\u0026#39;); 47 } else if (line.endsWith(\u0026#34; \u0026#34;)) { 48 // If line ends with a space, append \u0026#39;0\u0026#39; 49 binaryDataBuilder.append(\u0026#39;0\u0026#39;); 50 } 51 } 52 scanner.close(); 53 return binaryDataBuilder.toString(); 54 } 55 } This code demonstrates a basic form of text steganography where a secret message is hidden within the whitespace of a text document. Note that this method needs to be revised and may not be suitable for high-security applications. Advanced techniques can involve more sophisticated manipulation of text or embedding data within specific patterns.\nImage Steganography: # Data is concealed within digital images by modifying the least significant bits of pixel values or by embedding data within specific regions of the image where slight alterations are less noticeable.\nCertainly! Here\u0026rsquo;s a simple example of image steganography in Java using LSB (Least Significant Bit) embedding. In this example, we\u0026rsquo;ll hide a secret message within the least significant bits of an image\u0026rsquo;s pixels.\nThis code hides a secret message within the least significant bit of each pixel in the image. The original image is saved as a new steganographic image when the message is hidden. Later, the hidden message is extracted by reading the steganographic image\u0026rsquo;s pixels and retrieving the least significant bit of the red component of each pixel. Finally, the binary message is converted back into a readable string. Replace \u0026ldquo;input_image.png\u0026rdquo; with the path to your input image.\npublic class ImageSteganography { 8 9 // Method to encode a message into an image 10 public static BufferedImage encodeMessage(BufferedImage image, String message) { 11 int messageLength = message.length(); 12 int imageWidth = image.getWidth(); 13 int imageHeight = image.getHeight(); 14 int[] imagePixels = new int[imageWidth * imageHeight]; 15 image.getRGB(0, 0, imageWidth, imageHeight, imagePixels, 0, imageWidth); 16 17 // Convert the message into a binary string 18 StringBuilder binaryMessage = new StringBuilder(); 19 for (char character : message.toCharArray()) { 20 binaryMessage.append(String.format(\u0026#34;%8s\u0026#34;, Integer.toBinaryString(character)).replaceAll(\u0026#34; \u0026#34;, \u0026#34;0\u0026#34;)); 21 } 22 23 // Encode the message length at the beginning 24 String messageLengthBinary = String.format(\u0026#34;%32s\u0026#34;, Integer.toBinaryString(messageLength)).replace(\u0026#39; \u0026#39;, \u0026#39;0\u0026#39;); 25 binaryMessage.insert(0, messageLengthBinary); 26 27 // Encode the binary message into the image 28 for (int i = 0; i \u0026lt; binaryMessage.length(); i++) { 29 int pixel = imagePixels[i]; 30 int blue = pixel \u0026amp; 0xFF; 31 int green = (pixel \u0026gt;\u0026gt; 8) \u0026amp; 0xFF; 32 int red = (pixel \u0026gt;\u0026gt; 16) \u0026amp; 0xFF; 33 int alpha = (pixel \u0026gt;\u0026gt; 24) \u0026amp; 0xFF; 34 35 // Modify the LSB of the blue part of the pixel to match the current bit of the message 36 blue = (blue \u0026amp; 0xFE) | (binaryMessage.charAt(i) - \u0026#39;0\u0026#39;); 37 int newPixel = (alpha \u0026lt;\u0026lt; 24) | (red \u0026lt;\u0026lt; 16) | (green \u0026lt;\u0026lt; 8) | blue; 38 39 imagePixels[i] = newPixel; 40 } 41 42 BufferedImage newImage = new BufferedImage(imageWidth, imageHeight, BufferedImage.TYPE_INT_ARGB); 43 newImage.setRGB(0, 0, imageWidth, imageHeight, imagePixels, 0, imageWidth); 44 return newImage; 45 } 46 // Method to decode the hidden message from an image 47 public static String decodeMessage(BufferedImage image) { 48 int imageWidth = image.getWidth(); 49 int imageHeight = image.getHeight(); 50 int[] imagePixels = new int[imageWidth * imageHeight]; 51 image.getRGB(0, 0, imageWidth, imageHeight, imagePixels, 0, imageWidth); 52 53 // Extract the length of the message 54 StringBuilder messageLengthBinary = new StringBuilder(); 55 for (int i = 0; i \u0026lt; 32; i++) { 56 int pixel = imagePixels[i]; 57 int blue = pixel \u0026amp; 0xFF; 58 messageLengthBinary.append(blue \u0026amp; 1); 59 } 60 int messageLength = Integer.parseInt(messageLengthBinary.toString(), 2); 61 62 // Extract the binary message from the image 63 StringBuilder binaryMessage = new StringBuilder(); 64 for (int i = 32; i \u0026lt; 32 + messageLength * 8; i++) { 65 int pixel = imagePixels[i]; 66 int blue = pixel \u0026amp; 0xFF; 67 binaryMessage.append(blue \u0026amp; 1); 68 } 69 70 // Convert the binary message to string 71 StringBuilder message = new StringBuilder(); 72 for (int i = 0; i \u0026lt; binaryMessage.length(); i += 8) { 73 String byteString = binaryMessage.substring(i, i + 8); 74 int charCode = Integer.parseInt(byteString, 2); 75 message.append((char) charCode); 76 } 77 78 return message.toString(); 79 } 80 public static void main(String[] args) throws IOException { 81 File originalImageFile = new File(\u0026#34;_data/_DSC1259.png\u0026#34;); // Specify the path to the input image 82 BufferedImage originalImage = ImageIO.read(originalImageFile); 83 String secretMessage = \u0026#34;Secret message goes here\u0026#34;; 84 85 // Encode the message into the image 86 BufferedImage encodedImage = encodeMessage(originalImage, secretMessage); 87 88 // Save the encoded image 89 File outputImageFile = new File(\u0026#34;_data/_DSC1259_with-data.png\u0026#34;); // Specify the path to the output image 90 ImageIO.write(encodedImage, \u0026#34;png\u0026#34;, outputImageFile); 91 System.out.println(\u0026#34;The message has been encoded into the image.\u0026#34;); 92 93 94 File imageFile = new File(\u0026#34;_data/_DSC1259_with-data.png\u0026#34;); // Specify the path to the encoded image 95 BufferedImage image = ImageIO.read(imageFile); 96 97 // Decode the message from the image 98 String decodedMessage = decodeMessage(image); 99 System.out.println(\u0026#34;The hidden message is: \u0026#34; + decodedMessage); 100 calculatingMaxInfoAmount(\u0026#34;_data/_DSC1259_with-data.png\u0026#34;); 101 } 102 private static void calculatingMaxInfoAmount(String pathname){ 103 try { 104 File imageFile = new File(pathname); // Specify the path to your image 105 BufferedImage image = ImageIO.read(imageFile); 106 int imageWidth = image.getWidth(); 107 int imageHeight = image.getHeight(); 108 long maxBits = (long) imageWidth * imageHeight; // Maximum bits that can be stored 109 long maxBytes = maxBits / 8; // Convert bits to bytes 110 111 System.out.println(\u0026#34;Maximum information that can be stored in bytes: \u0026#34; + maxBytes); 112 } catch (IOException e) { 113 throw new RuntimeException(e); 114 } 115 } 116 } Audio Steganography: # Hiding information within audio files by modifying the least significant bits of audio samples or by exploiting imperceptible frequencies.\nHere\u0026rsquo;s a basic example of audio steganography in Java using LSB (Least Significant Bit) embedding. In this example, we\u0026rsquo;ll hide a secret message within the least significant bits of the audio samples.\npublic class AudioSteganography { 6 7 public static void main(String[] args) { 8 File inputFile = new File(\u0026#34;_data/Security - 2024.06 - What is Steganography-16bit-single-track_A01_L.wav\u0026#34;); 9 File outputFile = new File(\u0026#34;_data/output.wav\u0026#34;); 10 String message = \u0026#34;Secret Message\u0026#34;; 11 hideMessage(inputFile, outputFile, message); 12 13 File inputFileEncoded = new File(\u0026#34;_data/output.wav\u0026#34;); // This is the file with the hidden message 14 String extractedMessage = extractMessage(inputFileEncoded, 14); // Assuming we know the message length 15 System.out.println(\u0026#34;Extracted Message: \u0026#34; + extractedMessage); 16 } 17 18 public static void hideMessage(File inputFile, File outputFile, String message) { 19 try { 20 // Convert the message to a binary string 21 byte[] messageBytes = message.getBytes(); 22 StringBuilder binaryMessage = new StringBuilder(); 23 for (byte b : messageBytes) { 24 binaryMessage.append(String.format(\u0026#34;%8s\u0026#34;, Integer.toBinaryString(b \u0026amp; 0xFF)).replace(\u0026#39; \u0026#39;, \u0026#39;0\u0026#39;)); 25 } 26 String messageBinary = binaryMessage.toString(); 27 28 // Load the audio file 29 AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(inputFile); 30 AudioFormat format = audioInputStream.getFormat(); 31 byte[] audioBytes = audioInputStream.readAllBytes(); 32 33 // Hide the message in the LSB of the audio bytes 34 int messageIndex = 0; 35 for (int i = 0; i \u0026lt; audioBytes.length \u0026amp;\u0026amp; messageIndex \u0026lt; messageBinary.length(); i += 2) { // Skip every other byte for 16-bit samples 36 if (messageBinary.charAt(messageIndex) == \u0026#39;1\u0026#39;) { 37 audioBytes[i] = (byte) (audioBytes[i] | 1); // Set LSB to 1 38 } else { 39 audioBytes[i] = (byte) (audioBytes[i] \u0026amp; ~1); // Set LSB to 0 40 } 41 messageIndex++; 42 } 43 44 // Write the modified samples to a new file 45 ByteArrayInputStream bais = new ByteArrayInputStream(audioBytes); 46 AudioInputStream outputAudioInputStream = new AudioInputStream(bais, format, audioBytes.length / format.getFrameSize()); 47 AudioSystem.write(outputAudioInputStream, AudioFileFormat.Type.WAVE, outputFile); 48 49 System.out.println(\u0026#34;The message has been hidden in \u0026#34; + outputFile.getName()); 50 51 } catch (UnsupportedAudioFileException | IOException e) { 52 e.printStackTrace(); 53 } 54 } 55 56 public static String extractMessage(File inputFile, int messageLength) { 57 try { 58 // Load the audio file 59 AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(inputFile); 60 byte[] audioBytes = audioInputStream.readAllBytes(); 61 62 // Extract bits to reconstruct the message binary string 63 StringBuilder messageBinary = new StringBuilder(); 64 for (int i = 0; i \u0026lt; messageLength * 8 * 2; i += 2) { // Assuming 16-bit samples, adjust for actual sample size 65 byte b = audioBytes[i]; 66 int lsb = b \u0026amp; 1; // Extract the LSB 67 messageBinary.append(lsb); 68 } 69 70 // Convert the binary string to text 71 StringBuilder message = new StringBuilder(); 72 for (int i = 0; i \u0026lt; messageBinary.length(); i += 8) { 73 String byteString = messageBinary.substring(i, i + 8); 74 int charCode = Integer.parseInt(byteString, 2); 75 message.append((char) charCode); 76 } 77 78 return message.toString(); 79 80 } catch (UnsupportedAudioFileException | IOException e) { 81 e.printStackTrace(); 82 } 83 84 return null; 85 } 86 } This code hides a secret message within the input audio file\u0026rsquo;s least significant bit of each audio sample. The modified audio data is then saved as a new steganographic audio file. Later, the hidden message is extracted by reading the least significant bit of each audio sample in the steganographic audio file. Replace filename and path with the path to your input audio file.\nVideo Steganography: # Concealing data within video files by manipulating frames or embedding data in specific segments.\nBelow is a basic example of video steganography in Java using LSB (Least Significant Bit) embedding. In this example, we\u0026rsquo;ll hide a secret message within the least significant bits of the blue pixel values of video frames.\npublic class VideoSteganography { 14 //Should be extracted from the orig video 15 public static final int FPS = 50; 16 public static final String INPUT_FILE = \u0026#34;input.mp4\u0026#34;; 17 public static final String OUTPUT_FILE = \u0026#34;output.mp4\u0026#34;; 18 19 public static void main(String[] args) throws Exception { 20 FileChannelWrapper fileChannelWrapperIN = NIOUtils.readableChannel(new File(INPUT_FILE)); 21 File outputFile = new File(OUTPUT_FILE); 22 // Create a SequenceEncoder for the output file at 25 frames per second 23 SequenceEncoder encoder = SequenceEncoder.createSequenceEncoder(outputFile, FPS); 24 FrameGrab grab = FrameGrab.createFrameGrab(fileChannelWrapperIN); 25 Picture picture; 26 // to improve to process 27 // 0. detect the FPS from the input video 28 // 1. calculate the max amount of possible bytes that can be encoded 29 // 2. calculate the max amount of bytes that can be stored in each frame 30 // 3. split the message into chunks to fit into a frame 31 // 4. define a sequence to mark the end of the message 32 33 while (null != (picture = grab.getNativeFrame())) { 34 // Here, convert the Picture to BufferedImage 35 BufferedImage frame = AWTUtil.toBufferedImage(picture); 36 // Modify the frame to encode part of the message 37 BufferedImage encodedMessageOnBlue = encodeMessageOnBlue(frame, toBinaryString(\u0026#34;Hello Message\u0026#34;)); 38 // Convert BufferedImage to Picture (required by JCodec) 39 Picture pic = AWTUtil.fromBufferedImage(encodedMessageOnBlue, ColorSpace.RGB); 40 // Encode the modified frame back into the video 41 encoder.encodeNativeFrame(pic); 42 } 43 // Finalize video encoding and clean up 44 NIOUtils.closeQuietly(fileChannelWrapperIN); 45 // Finalize and close the encoder (important!) 46 encoder.finish(); 47 } 48 49 private static String toBinaryString(String message) { 50 StringBuilder binary = new StringBuilder(); 51 for (char character : message.toCharArray()) { 52 binary.append(String 53 .format(\u0026#34;%8s\u0026#34;, Integer.toBinaryString(character)) 54 .replace(\u0026#39; \u0026#39;, \u0026#39;0\u0026#39;)); 55 } 56 return binary.toString(); 57 } 58 59 private static BufferedImage encodeMessageOnBlue(BufferedImage image, String binaryMessage) { 60 int messageIndex = 0; 61 int messageLength = binaryMessage.length(); 62 63 for (int y = 0; y \u0026lt; image.getHeight() \u0026amp;\u0026amp; messageIndex \u0026lt; messageLength; y++) { 64 for (int x = 0; x \u0026lt; image.getWidth() \u0026amp;\u0026amp; messageIndex \u0026lt; messageLength; x++) { 65 int pixel = image.getRGB(x, y); 66 int alpha = (pixel \u0026gt;\u0026gt; 24) \u0026amp; 0xFF; 67 int red = (pixel \u0026gt;\u0026gt; 16) \u0026amp; 0xFF; 68 int green = (pixel \u0026gt;\u0026gt; 8) \u0026amp; 0xFF; 69 int blue = pixel \u0026amp; 0xFF; 70 71 // Modify the LSB of the blue component 72 blue = (blue \u0026amp; 0xFE) | (binaryMessage.charAt(messageIndex) - \u0026#39;0\u0026#39;); 73 messageIndex++; 74 75 // Reconstruct the pixel and set it back 76 int newPixel = (alpha \u0026lt;\u0026lt; 24) | (red \u0026lt;\u0026lt; 16) | (green \u0026lt;\u0026lt; 8) | blue; 77 image.setRGB(x, y, newPixel); 78 } 79 } 80 return image; 81 } 82 } File Steganography: # Embedding files within other files, such as hiding a document within another document.\nFile steganography with PDF files involves hiding one PDF file within another PDF file. In this example, we\u0026rsquo;ll hide the contents of one PDF file within the metadata of another PDF file. Specifically, we\u0026rsquo;ll utilise the \u0026ldquo;Keywords \u0026quot; metadata field of PDFs to embed the content of a secret PDF file.\nHere\u0026rsquo;s the Java code to achieve this using the Apache PDFBox library:\n9 10 public class HideMessageInPDF { 11 public static void main(String[] args) { 12 try { 13 // Load an existing PDF document 14 File file = new File(\u0026#34;_data/ReadME.pdf\u0026#34;); 15 PDDocument document = PDDocument.load(file); 16 // Retrieve the document\u0026#39;s metadata 17 PDDocumentInformation info = document.getDocumentInformation(); 18 // Add a hidden message to the metadata 19 // You could use a less obvious key than \u0026#34;HiddenMessage\u0026#34; to make it less detectable 20 info.setCustomMetadataValue(\u0026#34;HiddenMessage\u0026#34;, \u0026#34;This is a secret message\u0026#34;); 21 // Save the modified document 22 document.save(\u0026#34;_data/ReadME_with-data.pdf\u0026#34;); 23 document.close(); 24 System.out.println(\u0026#34;Hidden message added to the PDF metadata.\u0026#34;); 25 } catch (IOException e) { 26 e.printStackTrace(); 27 } 28 29 try { 30 // Load the PDF document 31 File file = new File(\u0026#34;_data/ReadME_with-data.pdf\u0026#34;); 32 PDDocument document = PDDocument.load(file); 33 // Access the document\u0026#39;s metadata 34 PDDocumentInformation info = document.getDocumentInformation(); 35 // Retrieve the hidden message from the metadata 36 String hiddenMessage = info.getCustomMetadataValue(\u0026#34;HiddenMessage\u0026#34;); 37 System.out.println(\u0026#34;Hidden message: \u0026#34; + hiddenMessage); 38 document.close(); 39 } catch (IOException e) { 40 e.printStackTrace(); 41 } 42 } 43 } In this code:\n1. The \u0026ldquo;hidePDF \u0026quot; function loads the source PDF document, converts the content of the secret PDF file to a Base64 encoded string, and embeds it into the \u0026ldquo;Keywords \u0026quot; metadata field of the source PDF file.\n2. The \u0026ldquo;extractPDF \u0026quot; function loads the steganographic PDF document, extracts the Base64 encoded content from the \u0026ldquo;Keywords \u0026quot; metadata field, decodes it, and writes it to an output PDF file.\nMake sure to replace every path to files with the appropriate file paths in your system. Additionally, you\u0026rsquo;ll need to include the Apache PDFBox library in your project to use its functionalities.\nConclusion: # As explored in this paper, steganography is a fascinating field with a rich history and diverse applications. By concealing information from ordinary data, steganography provides a powerful means of covert communication and data protection. Steganography continues to evolve alongside technological advancements, from ancient techniques of hiding messages in wax tablets to modern digital methods embedded within multimedia files.\nWhile steganography offers numerous benefits in fields like digital watermarking, copyright protection, and secure communication, it also presents challenges and ethical considerations. Due to its subtle nature, detecting steganographic content remains a significant challenge, requiring advanced algorithms and tools for effective analysis.\nAs technology continues to advance, the importance of understanding steganography becomes increasingly critical. It underscores the need for robust security measures balanced with respect for privacy rights. By delving into the principles and techniques of steganography, researchers and practitioners can develop more effective strategies for exploiting and defending against covert communication methods.\nIn conclusion, steganography represents a dynamic and intriguing aspect of information security, with implications spanning various domains. By embracing its complexities and exploring its potential applications, we can navigate the intricate landscape of digital communication with greater insight and resilience.\n","date":"28 March 2024","externalUrl":null,"permalink":"/posts/beyond-the-visible-exploring-the-depths-of-steganography/","section":"Posts","summary":"Steganography is the practice of concealing a message, file, image, or video within another message, file, image, or video. Unlike cryptography, which focuses on making a message unreadable to unauthorised parties, steganography aims to hide the message’s existence. The word “steganography \" is derived from the Greek words “steganos ,” meaning “covered ,” and “graphein ,” meaning “to write.”\n","title":"Beyond the Visible: Exploring the Depths of Steganography","type":"posts"},{"content":"","date":"6 March 2024","externalUrl":null,"permalink":"/tags/carinthia/","section":"Tags","summary":"","title":"Carinthia","type":"tags"},{"content":"","date":"6 March 2024","externalUrl":null,"permalink":"/tags/d1200x/","section":"Tags","summary":"","title":"D1200x","type":"tags"},{"content":"","date":"6 March 2024","externalUrl":null,"permalink":"/tags/sleepingbag/","section":"Tags","summary":"","title":"Sleepingbag","type":"tags"},{"content":" What is a winter trekking tour? # A winter trekking tour typically involves hiking or walking through snow-covered landscapes during the winter season. These tours are popular in regions with significant snowfall and mountainous terrain, such as the Alps, the Rockies, the Himalayas, and other mountain ranges worldwide. Winter trekking tours offer adventurers the opportunity to explore stunning winter landscapes, experience the tranquillity of the snowy wilderness, and challenge themselves physically and mentally.\nDuring a winter trekking tour, participants usually traverse trails or routes that may be inaccessible or less frequented during other seasons. Along the way, they may encounter snowy forests, frozen lakes, snow-capped peaks, and breathtaking vistas. Depending on the difficulty level of the trek, participants may need to navigate through various terrains, including deep snow, icy patches, and steep slopes.\nhttps://youtu.be/oKL7j7OuADk\nWinter trekking tours can vary in duration and intensity, ranging from single-day excursions to multi-day expeditions. Some tours may involve camping in tents, mountain huts, or lodges along the route. Participants also need specialised gear such as insulated clothing, waterproof boots, trekking poles, and sometimes snowshoes or crampons for traction on icy surfaces.\nOverall, winter trekking tours offer a unique and exhilarating outdoor experience for those who enjoy adventure and the beauty of winter landscapes. However, they also require careful planning, preparation, and awareness of the potential challenges and risks associated with winter hiking.\nWhat is a winter trekking tour? How important is a suitable sleeping bag? Insulation: Comfort: Weight and Packability: Durability: Versatility: What are the isolation materials for Winter-Sleeping-Bags? Down Insulation: Synthetic Insulation: What are the cons of Down Isolation? Price: Maintenance: Water Sensitivity: Allergies: Longer Drying Time: Ethical Concerns: What are the cons of synthetic Isolation? Bulk and Weight: Less Compressible: Lower Warmth-to-Weight Ratio: Reduced Durability: Limited Breathability: Environmental Impact: Flammability: How is the Isolation working with Downs? What Down is best for Isolation? Are all Goose Downs good? Responsible Down Standard (RDS) and the Global Traceable Down Standard (Global TDS) Responsible Down Standard (RDS): Global Traceable Down Standard (Global TDS): Is there any European or German standard? 1. Downpass: 2. Textile Exchange\u0026rsquo;s Europe-Based Initiatives: 3. German Legislation and Animal Welfare Standards: How do you maintain Down insulation in sleeping bags? How important is a suitable sleeping bag? # A good sleeping bag is significant for any outdoor adventure, especially for camping, trekking, or backpacking, where you\u0026rsquo;ll be spending nights outdoors. Here\u0026rsquo;s why:\nInsulation: # A quality sleeping bag provides insulation to keep you warm during cold nights. Insulation is usually measured in fill power (for down sleeping bags) or temperature ratings (for both down and synthetic sleeping bags). The higher the fill power or the lower the temperature rating, the better the insulation.\nComfort: # A well-designed sleeping bag ensures comfort by providing adequate space to move around while fitting snugly to retain heat. Features like draft tubes, hood cinches, and zipper guards enhance comfort by preventing cold drafts and ensuring a cosy fit.\nWeight and Packability: # For outdoor activities where you need to carry your gear, a sleeping bag\u0026rsquo;s weight and packability are crucial factors. High-quality materials and efficient designs can significantly reduce a sleeping bag\u0026rsquo;s weight and volume, making it easier to pack and carry.\nDurability: # A good sleeping bag is durable and built to withstand the rigours of outdoor use. Quality construction, reinforced seams, and durable materials ensure that your sleeping bag will last through many adventures.\nVersatility: # Depending on the season and location of your outdoor adventure, you may need a sleeping bag with specific temperature ratings. Some sleeping bags are designed for use in particular conditions, while others offer versatility by providing options for temperature regulation, such as zippered vents or removable liners.\nOverall, a suitable sleeping bag is essential for ensuring a comfortable and enjoyable outdoor experience, whether camping in the backcountry, trekking through the wilderness, or simply spending a night under the stars. It\u0026rsquo;s worth researching and choosing a sleeping bag that fits your needs and provides the necessary insulation, comfort, and durability for your adventures.\nWhat are the isolation materials for Winter-Sleeping-Bags? # Winter sleeping bags require effective insulation to keep you warm in cold temperatures. The two primary insulation materials used in winter sleeping bags are down and synthetic. Here\u0026rsquo;s an overview of each:\nDown Insulation: # Goose or Duck Down : Down insulation is made from the delicate, soft clusters found under the feathers of geese or ducks. It provides exceptional warmth-to-weight ratio, compressibility, and durability.\nFill Power : Down insulation is measured in fill power, which indicates its loftiness and insulating ability. Higher fill power down provides better insulation with less weight.\nWater Resistance : One drawback of down is that it loses its insulating properties when wet. However, some manufacturers treat down with water-repellent coatings or use hydrophobic down to enhance water resistance.\nSynthetic Insulation: # Polyester Fibers : Synthetic insulation is made from polyester fibres designed to mimic the insulating properties of down. Unlike down, synthetic insulation retains its warmth even when wet, making it a more reliable choice in damp conditions.\nDurability : Synthetic insulation tends to be more durable and resilient than down, maintaining its loft and insulating properties even after repeated compression and exposure to moisture.\nAffordability : Synthetic sleeping bags are often more affordable than down sleeping bags, making them a popular choice for budget-conscious adventurers.\nBoth down and synthetic insulation have pros and cons, and the choice between them depends on factors such as your budget, the expected conditions of your trip, and personal preferences.\nWhat are the cons of Down Isolation? # While down insulation offers excellent warmth-to-weight ratio and compressibility, there are some drawbacks to consider:\nPrice: # Down sleeping bags tend to be more expensive than synthetic options. High-quality Down, especially those with higher fill power ratings, can significantly increase the cost of the sleeping bag.\nMaintenance: # Down requires careful maintenance to retain its loft and insulation properties. It can lose effectiveness if it becomes wet, clumps together, or gets compressed for extended periods without proper storage.\nWater Sensitivity: # Down loses its insulation properties when it gets wet. Moisture from rain, snow, or body sweat can reduce the loft of down, leading to decreased warmth and longer drying times. However, some manufacturers treat down with water-repellent coatings or use hydrophobic down to mitigate this issue.\nAllergies: # Some people may be allergic to down or sensitive to the dust and particles that can be present in down insulation. Synthetic insulation may be a better option for individuals with allergies or sensitivities.\nLonger Drying Time: # If a down sleeping bag becomes wet, it can take longer to dry than synthetic insulation. This can be problematic in wet or humid conditions, especially on an extended outdoor trip without access to drying facilities.\nEthical Concerns: # Some individuals may have ethical concerns about sourcing down insulation, particularly regarding animal welfare and practices such as live-plucking or force-feeding. Opting for responsibly sourced down or synthetic insulation can alleviate these concerns.\nDespite these drawbacks, down insulation remains famous for outdoor enthusiasts due to its superior warmth, compressibility, and longevity when correctly cared for. However, weighing these factors against your specific needs and preferences is essential when choosing a winter sleeping bag.\nWhat are the cons of synthetic Isolation? # While synthetic insulation has its advantages, there are also some drawbacks to consider:\nBulk and Weight: # Synthetic insulation tends to be heavier than down insulation, especially when packed into a sleeping bag. This can make synthetic sleeping bags less convenient for backpacking or other activities where weight and pack size are critical factors.\nLess Compressible: # Synthetic insulation doesn\u0026rsquo;t compress as well as down, so synthetic sleeping bags may take up more space in your pack compared to down sleeping bags with similar temperature ratings.\nLower Warmth-to-Weight Ratio: # Synthetic insulation generally has a lower warmth-to-weight ratio than down, meaning you may need a heavier sleeping bag to achieve the same level of warmth as a down sleeping bag.\nReduced Durability: # Over time and with repeated use, synthetic insulation can lose its loft and insulating properties more quickly than down. This can result in a shorter lifespan for synthetic sleeping bags than down sleeping bags.\nLimited Breathability: # Synthetic insulation may not breathe as well as down, leading to potential moisture buildup inside the sleeping bag. This can cause discomfort and contribute to a clammy feeling at night.\nEnvironmental Impact: # Many synthetic insulation materials are derived from petroleum-based products, which raises environmental concerns regarding their production and disposal. However, eco-friendly synthetic insulation options are also available that use recycled materials or sustainable manufacturing processes.\nFlammability: # Some synthetic insulation materials are more flammable than down, which may pose a safety risk in certain situations. However, many sleeping bags are treated with flame-retardant chemicals to mitigate this risk.\nDespite these drawbacks, synthetic insulation remains popular for many outdoor enthusiasts, especially those who prioritise affordability, moisture resistance, and ethical considerations. When choosing a winter sleeping bag, it\u0026rsquo;s essential to weigh the pros and cons of synthetic insulation against your specific needs and preferences.\nHow is the Isolation working with Downs? # Down insulation works by trapping air within the lofted clusters of down feathers, creating a barrier that prevents heat loss from your body to the surrounding environment. Here\u0026rsquo;s how the insulation process works:\nLoft : Down feathers have a three-dimensional structure with tiny filaments that interlock to create air pockets. This structure allows down clusters to loft or fluff up, creating a high volume of dead air space within the sleeping bag.\nDead Air Space : The dead air space created by the lofted-down clusters acts as an insulating layer. Air is an excellent insulator because it has low thermal conductivity, meaning it doesn\u0026rsquo;t transfer heat well. By trapping air within the down clusters, down insulation minimises heat transfer and helps maintain your body\u0026rsquo;s warmth.\nFill Power : Fill power is a measure of the loftiness or fluffiness of down insulation. Higher fill power down contains larger, more resilient clusters that trap more air and provide better insulation. Fill power is typically measured in cubic inches per ounce (in³/oz), with higher numbers indicating higher quality down.\nFluffing : To maximise the insulating properties of down insulation, it\u0026rsquo;s essential to fluff up the sleeping bag periodically to restore its loft. Fluffing redistributes the down clusters and prevents them from compressing over time, ensuring optimal warmth and insulation.\nWater Resistance : Down insulation is vulnerable to moisture because wet down loses its loft and insulating ability. However, some manufacturers treat down with water-repellent coatings or use hydrophobic down, specially treated to repel moisture and maintain its loft even when wet.\nOverall, down insulation is highly effective at trapping air and providing warmth in cold conditions. Its lightweight, compressible nature makes it ideal for winter sleeping bags, where warmth-to-weight ratio and packability are essential considerations. However, it\u0026rsquo;s critical to protect down insulation from moisture and maintain its loft to ensure optimal performance over time.\nWhat Down is best for Isolation? # When considering down insulation for sleeping bags, jackets, or other outdoor gear, the quality of the Down is a crucial factor. The quality of the Down is often measured by its fill power, which indicates the loftiness and insulating ability of the Down. Higher fill power down generally offers better insulation per ounce and is considered superior for outdoor use. Here\u0026rsquo;s a breakdown of common down qualities:\nFill Power : Fill power refers to the volume (in cubic inches) of one ounce of downfills when allowed to expand fully under specific conditions. Down with a higher fill power has larger and more resilient clusters, providing better loft and insulation. Common fill power ratings range from around 500 to 900 or more. Down with a fill power of 800 or higher is considered excellent quality and is often used in premium outdoor gear.\n800+ Fill Power : Down with a fill power of 800 or higher, it is considered top-of-the-line and offers exceptional warmth-to-weight ratio, compressibility, and loft. It provides excellent insulation for cold weather conditions while remaining lightweight and packable.\n700-800 Fill Power : Down with fill power, this range still offers excellent insulation and is suitable for most outdoor activities, including backpacking, camping, and mountaineering. It provides a good balance of warmth, weight, and affordability.\n600-700 Fill Power : Down in this range is considered mid-range and offers decent insulation for general outdoor use. While not as lofty or compressible as higher fill power down, it still provides adequate warmth for mild to moderate cold conditions.\n500-600 Fill Power : Down with fill power below 600 is considered lower quality and may be less efficient in providing insulation. It is often used in budget-friendly gear where weight and packability are less concerned.\nIn summary, the best Down for insulation depends on the specific requirements of your outdoor activities, budget, and personal preferences. For premium performance and lightweight gear, opt for down with a fill power of 800 or higher. However, lower fill power can still offer adequate insulation for less demanding activities at a more affordable price point.\nAre all Goose Downs good? # Not all Goose Down is created equal when it comes to insulation. While Goose Down, in general, is known for its excellent insulation properties, there can still be variations in quality based on factors such as the type of Goose, the maturity of the down clusters, and the processing methods used. Here are some factors to consider to ensure you\u0026rsquo;re getting the best insulation from Goose Down:\nType of Goose : Different species of geese produce down with varying qualities. Eiderdown, for example, is considered the finest and most luxurious type of Goose down, known for its exceptional loft, warmth, and lightness. However, it is also the most expensive. Other types of Goose down, such as white Goose down or grey Goose down, can still offer excellent insulation properties when properly processed and treated.\nProcessing Methods : How Goose Down is cleaned, sorted, and treated can affect insulation performance. High-quality Down undergoes thorough cleaning to remove dirt, dust, and oils while preserving the loft and resilience of the clusters. Down that needs to be better processed may have diminished loft and insulation capabilities.\nEthical Sourcing : Responsible sourcing of Goose Down is essential for many outdoor enthusiasts. Look for ethically sourced products from farms that adhere to animal welfare standards and sustainable practices. Responsible sourcing ensures that the geese are treated humanely and that the Down is obtained in an ethical and environmentally friendly manner.\nCertifications : Look for products certified by reputable organisations such as the Responsible Down Standard (RDS) or the Global Traceable Down Standard (Global TDS). These certifications ensure that the down used in the product is ethically sourced and traceable throughout the supply chain.\nIn summary, while Goose Down is generally good for insulation, factors such as fill power, type of Goose, processing methods, and ethical sourcing must be considered to ensure the best insulation performance from your own products.\nResponsible Down Standard (RDS) and the Global Traceable Down Standard (Global TDS) # The Responsible Down Standard (RDS) and the Global Traceable Down Standard (Global TDS) are two certification programs to ensure the ethical sourcing and traceability of down used in outdoor gear and apparel. Here\u0026rsquo;s more information about each standard:\nResponsible Down Standard (RDS): # The Responsible Down Standard is a global certification program developed by Textile Exchange, an international nonprofit organization focused on sustainability in the textile industry. The RDS aims to ensure that down and feathers used in products come from animals without unnecessary harm, such as live-plucking or force-feeding. The standard covers the entire supply chain, from farm to finished product, and includes requirements for animal welfare, traceability, and transparency. Certified RDS products must undergo independent audits by third-party certification bodies to verify compliance with the standard\u0026rsquo;s requirements. The RDS certification label provides consumers with assurance that the down used in the product meets strict ethical and welfare standards. Global Traceable Down Standard (Global TDS): # The Global Traceable Down Standard is another certification program developed to ensure the responsible sourcing and traceability of Down in outdoor products. The Global TDS was established by NSF International, a global public health and safety organisation, in collaboration with the outdoor industry. Like the RDS, the Global TDS focuses on animal welfare, traceability, and transparency throughout the Down supply chain. The standard requires that down used in certified products be traced back to the source farm or slaughterhouse, providing transparency and accountability. Certified Global TDS products must undergo regular audits and inspections by accredited certification bodies to ensure compliance with the standard\u0026rsquo;s requirements. The Global TDS certification label indicates that the down used in the product has been responsibly sourced and can be traced back to its origin. The Responsible Down Standard and the Global Traceable Down Standard assure consumers that the Down used in outdoor gear and apparel has been sourced ethically and transparently without causing unnecessary harm to animals. These certification programs are vital in promoting sustainability and animal welfare in the outdoor industry.\nIs there any European or German standard? # There are European standards and initiatives related to the responsible sourcing and traceability of down, although they may not be as widely recognised as the Responsible Down Standard (RDS) or the Global Traceable Down Standard (Global TDS). Here are a few examples:\nDownpass: # Downpass is a certification and traceability system for down and feathers developed by the Association of the European Bed Feather and Bedding Industries (EuroVets). Downpass aims to ensure the responsible sourcing and traceability of down and feathers used in bedding and outdoor products. The certification covers animal welfare, traceability, and quality assurance throughout the supply chain. Downpass-certified products undergo regular audits and inspections by accredited certification bodies to verify compliance with the standard\u0026rsquo;s requirements. Textile Exchange\u0026rsquo;s Europe-Based Initiatives: # Textile Exchange, the Responsible Down Standard (RDS) organisation, collaborates with European stakeholders to promote responsible down sourcing and traceability. While the RDS is a global standard, Textile Exchange works with European brands, retailers, and industry associations to implement responsible sourcing practices and support adopting certification programs such as the RDS. German Legislation and Animal Welfare Standards: # Germany, like other European countries, has legislation and regulations related to animal welfare that apply to the farming and handling of animals, including those used for down production. While these regulations may not specifically address down sourcing and traceability in outdoor products, they set standards for animal welfare that manufacturers and suppliers are expected to adhere to. Companies operating in Germany may also choose to adopt voluntary certifications or standards, such as Downpass or the RDS, to demonstrate their commitment to ethical sourcing and transparency. While there may not be a single European or German standard equivalent to the RDS or Global TDS, initiatives like Downpass and collaborations with organisations like Textile Exchange promote responsible down sourcing and ensure ethical practices in the European outdoor industry.\nHow do you maintain Down insulation in sleeping bags? # Maintaining down insulation in sleeping bags is essential to ensure their longevity, performance, and warmth. Here are some tips to help you properly care for and maintain down sleeping bags:\nStorage : Store your down sleeping bag uncompressed in a large cotton storage sack or hang it in a closet. Avoid storing it compressed for extended periods, which can cause the down clusters to lose loft and insulation effectiveness.\nAir Out : After each use, air out your sleeping bag to remove moisture and fully allow the Down to loft. Hang it in a well-ventilated area or drape it over a clothesline to facilitate airflow. Avoid exposing it to direct sunlight for extended periods, as this can degrade the fabric and insulation.\nUse a Liner : Consider using a sleeping bag liner to protect the interior of the sleeping bag from body oils, dirt, and sweat. Liners are easier to clean and can help extend the lifespan of your sleeping bag.\nSpot Clean : Spot clean any stains or dirt with a gentle detergent and lukewarm water. Use a soft brush or sponge to scrub the affected areas gently, then rinse thoroughly with clean water. Avoid using harsh chemicals, bleach, or fabric softeners, as they can damage the down and fabric.\nMachine Washing :\nIf your sleeping bag is machine washable, use a front-loading washing machine on a gentle cycle with a down-specific detergent. Avoid using top-loading machines with agitators, as they can damage the fabric and insulation. Ensure the sleeping bag is completely dry before storing it. Dry Cleaning : Some down sleeping bags may only be labelled as dry clean. If this is the case, take your sleeping bag to a professional cleaner experienced in handling down products. Ensure they use a down-specific cleaning process to maintain the loft and insulation.\nFluffing : Periodically fluff your sleeping bag by shaking it out and gently patting it to redistribute the down clusters evenly. This helps maintain loft and insulation effectiveness.\nAvoid Compression : When packing your sleeping bag for storage or travel, use a breathable storage sack or hang it loosely in a closet. Avoid compressing the sleeping bag for extended periods, which can cause the Down to lose loft and insulation properties.\nBy following these maintenance tips, you can help preserve your down sleeping bag\u0026rsquo;s loft, warmth, and longevity, ensuring years of comfortable outdoor adventures.\n","date":"6 March 2024","externalUrl":null,"permalink":"/posts/sleepingbags-for-winter-expeditions/","section":"Posts","summary":"What is a winter trekking tour? # A winter trekking tour typically involves hiking or walking through snow-covered landscapes during the winter season. These tours are popular in regions with significant snowfall and mountainous terrain, such as the Alps, the Rockies, the Himalayas, and other mountain ranges worldwide. Winter trekking tours offer adventurers the opportunity to explore stunning winter landscapes, experience the tranquillity of the snowy wilderness, and challenge themselves physically and mentally.\n","title":"Sleepingbags For Winter Expeditions","type":"posts"},{"content":"","date":"6 March 2024","externalUrl":null,"permalink":"/tags/trekking/","section":"Tags","summary":"","title":"Trekking","type":"tags"},{"content":"","date":"6 March 2024","externalUrl":null,"permalink":"/tags/winter/","section":"Tags","summary":"","title":"Winter","type":"tags"},{"content":"","date":"12 February 2024","externalUrl":null,"permalink":"/tags/compensating-transactions-pattern/","section":"Tags","summary":"","title":"Compensating Transactions Pattern","type":"tags"},{"content":"","date":"12 February 2024","externalUrl":null,"permalink":"/tags/distributed-design-pattern/","section":"Tags","summary":"","title":"Distributed Design Pattern","type":"tags"},{"content":" The Bird-Eye View # A Compensating Transaction Pattern is a technique used to ensure consistency when multiple steps are involved in a process, and some steps may fail. It essentially consists in having \u0026ldquo;undo\u0026rdquo; transactions for each successful step, so if something goes wrong later on, you can reverse the changes made earlier and maintain data integrity.\nHere\u0026rsquo;s a breakdown of the key points:\nWhat it does: # Ensures consistency in eventually consistent operations with multiple steps. Reverses the work done by previous successful steps if a later step fails. Maintains data integrity by undoing changes in case of failure. How it works: # Primary transaction: A series of steps are performed to complete a specific task. Compensating transactions: A \u0026ldquo;undo\u0026rdquo; transaction is designed and stored for each successful step. Failure: If a later step fails, the compensating transactions for the completed steps are executed reversely, effectively undoing the changes. The Bird-Eye View What it does: How it works: Benefits: Examples of use cases: Online Hotel Booking System: Inventory Management System: Money Transfer Service: Flight Reservation System: Order Processing System - (e-commerce): Tradeoffs: Complexity: Concurrency and Race Conditions: Performance Overhead: Overhead of Compensating Actions: Latency and Response Time: Concurrency and Scalability: Network and I/O Overhead: Transaction Retry and Rollback: Resource Consumption: Monitoring and Logging Overhead: Data Consistency: Error Handling and Recovery: Auditability and Monitoring: Development and Testing Complexity: Compensating Action Design: Testing Failure Scenarios: Concurrency and Race Conditions: Error Handling and Recovery Testing: Integration Testing: End-to-End Testing: Unsuccessful Transaction Example: Online Shopping Fail Main Transaction: Compensating Transactions: Failure: Compensating Actions: Outcome: Additional Notes: Implementing Compensating Transactions in Core Java (Simplified Example) What other Patterns are often combined? Retry Pattern: Saga Pattern: Event Sourcing Pattern: Circuit Breaker Pattern: Idempotent Receiver Pattern: Conclusion: Benefits: # The Compensating Transaction Pattern offers several benefits in the context of distributed systems and complex transactions:\nFault Tolerance : The pattern enhances fault tolerance by providing a mechanism to handle failures gracefully. When a failure occurs during a transaction, compensating actions can be executed to revert the system to a consistent state, ensuring that partial failures do not leave the system in an inconsistent or corrupted state.\nConsistency : Compensating transactions helps maintain data consistency across distributed systems by ensuring that changes made by failed transactions are properly reverted. This helps prevent data anomalies and ensures the system remains consistent, even during failures.\nTransaction Rollback : Unlike traditional transaction rollback mechanisms, compensating transactions offer more flexibility in handling complex transactions that involve multiple components or services. Instead of rolling back the entire transaction, compensating actions can selectively undo the effects of failed transactions, allowing other parts of the transaction to proceed unaffected.\nResilience : By incorporating compensating transactions into the system design, developers can improve the resilience of distributed systems against various types of failures, including network issues, service unavailability, or system crashes. This resilience contributes to overall system stability and reliability.\nFlexibility : The pattern provides flexibility in designing and implementing transactional workflows in distributed systems. Developers can define compensating actions tailored to specific transactional requirements, allowing for custom handling of failures and recovery scenarios.\nEnhanced Scalability : Compensating transactions can facilitate the implementation of scalable distributed systems by decoupling transactional logic from individual components or services. This decoupling allows for better scalability and parallelisation of transaction processing across distributed environments.\nImproved Error Handling : Compensating transactions enables more robust error handling mechanisms by providing a structured approach to handling transaction failures. Developers can implement specific error-handling logic within compensating actions to address various failure scenarios, such as resource constraints or service timeouts.\nOverall, the Compensating Transaction Pattern offers significant benefits in fault tolerance, consistency, resilience, flexibility, scalability, and error handling, making it a valuable pattern for designing and implementing distributed systems with complex transactional requirements.\nExamples of use cases: # Here are five examples of the Compensating Transaction Pattern in action across various domains:\nOnline Hotel Booking System: # Main Transaction : Reserve a hotel room for a customer and charge their credit card.\nCompensating Actions : If charging the credit card fails or the reservation cannot be made due to a system error, release the reserved room and void the transaction authorisation on the credit card.\nInventory Management System: # Main Transaction : Deduct inventory stock when an order is placed.\nCompensating Actions : If an order cannot be fulfilled due to insufficient stock or system errors, increment the stock count for the ordered items to revert the deduction.\nMoney Transfer Service: # Main Transaction : Transfer funds from one account to another.\nCompensating Actions : If the transfer fails due to a network issue or insufficient funds, reverse the transfer by depositing the amount back into the sender\u0026rsquo;s account.\nFlight Reservation System: # Main Transaction : Book a flight ticket for a passenger and debit the payment from their account.\nCompensating Actions : If the payment fails or the reservation cannot be completed, cancel the booking and refund the payment to the passenger\u0026rsquo;s account.\nOrder Processing System - (e-commerce): # Main Transaction : Process an order by updating inventory, charging the customer, and notifying the warehouse for shipment.\nCompensating Actions : If any step fails (e.g., payment authorisation fails, inventory update fails, or shipment notification fails), revert the changes made and inform the customer about the failure.\nIn each of these examples, the Compensating Transaction Pattern ensures that the system maintains consistency and integrity, even in the face of failures or errors during transaction processing. Compensating actions are designed to undo the effects of failed transactions and restore the system to a consistent state.\nTradeoffs: # While the Compensating Transaction Pattern offers several benefits, it also comes with inevitable tradeoffs and considerations that developers need to take into account:\nComplexity: # Implementing compensating transactions can add complexity to the system design and codebase. Developers must carefully design and manage compensating actions to ensure they accurately revert the effects of failed transactions. This complexity can make the codebase harder to understand, maintain, and debug.\nConcurrency and Race Conditions: # In distributed systems with concurrent transactions, coordinating compensating actions across multiple components or services can introduce race conditions and synchronisation challenges. Ensuring that compensating actions are executed correctly and idempotently in the presence of concurrent transactions requires careful attention to concurrency control mechanisms.\nPerformance Overhead: # Executing compensating actions to revert failed transactions incurs additional processing overhead and resource utilisation. Depending on the complexity of compensating actions and the frequency of transaction failures, this overhead can impact system performance and scalability, particularly in high-throughput environments.\nThe performance implications of the Compensating Transaction Pattern depend on various factors, including the complexity of the transactional workflow, the frequency of transaction failures, the efficiency of compensating actions, and the underlying infrastructure of the distributed system. Here are some performance considerations associated with this pattern:\nOverhead of Compensating Actions: # Executing compensating actions to revert failed transactions incurs additional processing overhead and resource utilisation. Depending on the complexity of compensating actions, their execution may involve database updates, service calls, or other operations that consume CPU, memory, and network bandwidth.\nLatency and Response Time: # The time required to execute compensating actions impacts the overall latency and response time of transaction processing. If compensating actions involve time-consuming operations or depend on external services, they can introduce delays in recovering from transaction failures, affecting system responsiveness and user experience.\nConcurrency and Scalability: # Coordinating compensating actions across concurrent transactions can introduce contention and synchronisation overhead, impacting system scalability. In highly concurrent environments, ensuring that compensating actions are executed correctly and efficiently without causing bottlenecks or resource contention is crucial for maintaining performance under load.\nNetwork and I/O Overhead: # Compensating actions involving remote services or accessing distributed data sources incur network and I/O overhead. Network latency, message serialisation/deserialisation, and network congestion can affect the performance of compensating transactions, particularly in geographically distributed systems.\nTransaction Retry and Rollback: # Implementing retry mechanisms for failed transactions and rolling back partially completed transactions involve additional computational and I/O overhead. Retrying failed transactions increases the workload on the system, primarily if retries are performed with exponential backoff or involve multiple retries before triggering compensating actions.\nResource Consumption: # Compensating transactions may consume additional resources such as memory, disk space, and database connections. Managing resource usage and contention, especially in multi-tenant or shared infrastructure environments, is essential for maintaining overall system performance and stability.\nMonitoring and Logging Overhead: # Capturing transactional events, logging execution outcomes, and monitoring the performance of compensating transactions impose overhead on system resources, particularly in environments with high transaction volumes. Balancing the granularity of monitoring and logging with performance considerations is essential for optimising system performance.\nTo mitigate the performance implications of the Compensating Transaction Pattern, developers can employ various strategies such as optimising compensating actions, reducing transactional complexity, implementing efficient retry mechanisms, optimising network communication, scaling infrastructure resources, and using caching and batching techniques. Additionally, conducting performance testing and profiling to identify and address performance bottlenecks is essential for optimising the performance of systems that utilise compensating transactions.\nData Consistency: # While compensating transactions helps maintain consistency in the face of transaction failures, ensuring data consistency across distributed systems remains a challenge. Coordinating compensating actions with other concurrent transactions and ensuring that data updates are properly synchronised can be complex, especially in systems with eventual consistency requirements.\nHere\u0026rsquo;s an example illustrating the potential data consistency problem:\nConsider an e-commerce platform where customers can place orders for products. When a customer places an order, the system deducts the ordered items from the inventory and charges the customer\u0026rsquo;s credit card. If either of these actions fails, compensating actions are executed to revert the transaction.\nLet\u0026rsquo;s suppose a customer orders a product, the inventory is successfully deducted, but the credit card charge fails due to a network issue. The compensating action would be adding the deducted items to the inventory. However, due to the system\u0026rsquo;s distributed nature, there\u0026rsquo;s a delay in executing the compensating action; in the meantime, another customer orders the same product.\nIf the compensating action to add back the deducted items is not synchronised or appropriately coordinated, it may lead to data inconsistency. Specifically, the inventory count may need to be corrected if the compensating action adds back the deducted items while another order for the same product is already processed. This can result in overselling products and accurate inventory counts, leading to customer satisfaction and operational inefficiencies.\nTo address this data consistency problem, developers need to implement proper synchronisation and coordination mechanisms to ensure that compensating actions are executed in a consistent and idempotent manner. This may involve using distributed transactions, locks, or other coordination techniques to coordinate compensating actions across distributed components and maintain data consistency. Additionally, implementing mechanisms to handle concurrent updates and conflicts, such as optimistic concurrency control or conflict resolution strategies, can help mitigate data consistency issues using the Compensating Transaction Pattern.\nError Handling and Recovery: # Implementing robust error handling and recovery mechanisms for compensating transactions is essential but challenging. Developers need to anticipate and handle various failure scenarios, including partial failures, network issues, and service unavailability, to ensure that compensating actions are executed reliably and effectively.\nAuditability and Monitoring: # Tracking and auditing compensating transactions for accountability and compliance purposes can be challenging, especially in complex distributed systems with multiple transactional workflows. Implementing comprehensive monitoring and logging mechanisms to capture transactional events and execution outcomes is crucial for troubleshooting and compliance.\nDevelopment and Testing Complexity: # Developing and testing compensating transactions require thoroughly validating the main transactional workflow and compensating actions. Testing failure scenarios and edge cases to ensure the correctness and robustness of compensating transactions can be time-consuming and resource-intensive.\nHere\u0026rsquo;s an example illustrating the complexities involved:\nConsider a scenario where you\u0026rsquo;re developing an online banking system that allows users to transfer funds between accounts. The system must deduct funds from the sender\u0026rsquo;s account and credit them to the recipient\u0026rsquo;s account. If the transfer fails (e.g., insufficient funds, network error), compensating actions must be executed to revert the transaction and restore the accounts to their original states.\nNow, during development and testing of this system, you encounter several challenges related to the Compensating Transaction Pattern:\nCompensating Action Design: # Designing compensating actions requires careful consideration of the actions needed to revert the effects of failed transactions. In the banking system example, you must define compensating actions to restore the sender\u0026rsquo;s account balance if the transfer fails. This may involve adding back the deducted funds and updating transaction logs or audit trails.\nTesting Failure Scenarios: # Testing the system under various failure scenarios to ensure that compensating actions are triggered and executed correctly adds complexity to the testing process. For example, you must simulate scenarios such as insufficient funds, network timeouts, and service failures to verify that compensating actions are invoked and revert the transaction as expected.\nConcurrency and Race Conditions: # Testing concurrent transactions and ensuring that compensating actions are executed correctly in a concurrent environment requires thorough testing and validation. Race conditions and synchronisation issues may arise when multiple transactions are processed concurrently, leading to inconsistent or incorrect compensating action execution.\nError Handling and Recovery Testing: # Validating error handling and recovery mechanisms to ensure that the system gracefully handles failures and recovers consistently adds to the testing complexity. This includes testing retry mechanisms, rollback procedures, and error recovery paths to verify that the system behaves as expected under failure conditions.\nIntegration Testing: # Testing the integration of compensating transactions with other system components and services introduces additional complexity. To maintain data consistency and integrity, you need to ensure that compensating actions interact correctly with external dependencies, such as databases, messaging systems, and third-party services.\nEnd-to-End Testing: # To validate the entire transactional workflow, including main transactions and compensating actions, end-to-end testing requires comprehensive testing scenarios and test coverage. This involves testing different transaction sequences, error scenarios, and edge cases to verify the correctness and robustness of the system.\nOverall, addressing the development and testing complexity associated with the Compensating Transaction Pattern requires thorough planning, design, and validation to ensure that compensating actions are correctly implemented, tested, and integrated into the system. Effective testing strategies, automation, and collaboration between development and testing teams are essential for mitigating the challenges and complexities of implementing this pattern.\nIn summary, while the Compensating Transaction Pattern offers benefits such as fault tolerance, consistency, and resilience, it also introduces complexity, concurrency challenges, performance overhead, and additional data consistency, error handling, auditability, and testing considerations. Understanding these tradeoffs and carefully designing compensating transactions is essential for effectively leveraging this pattern in distributed systems.\nUnsuccessful Transaction Example: Online Shopping Fail # Imagine you\u0026rsquo;re buying a new pair of shoes online. Here\u0026rsquo;s an example of how a Compensating Transaction Pattern could be used in this scenario, and how it would work if the transaction failed:\nSteps:\nMain Transaction: # You add the shoes to your cart and proceed to checkout. You enter your billing and shipping information. You authorise the payment for the shoes. The store\u0026rsquo;s inventory system updates, marking the shoes as \u0026ldquo;sold.\u0026rdquo; The order confirmation email is sent to you. Compensating Transactions: # For each successful step:\nA compensating transaction is designed and stored. For example, if needed, the inventory update would have a compensating transaction to reverse the \u0026ldquo;sold\u0026rdquo; status. Failure: # Let\u0026rsquo;s say the shipping address verification fails after successful payment authorisation. The main transaction cannot be completed.\nCompensating Actions: # The compensating transaction for the inventory update is triggered. The shoes are marked as \u0026ldquo;available\u0026rdquo; again. The payment authorisation is reversed, and the funds are returned to your account. You receive a notification about the failed transaction and the reason for failure. Outcome: # While the purchase wasn\u0026rsquo;t successful, the Compensating Transaction Pattern ensures data consistency. Your money is safe, and the store\u0026rsquo;s inventory remains accurate. Additional Notes: # This is a simplified example, and real-world implementations can be more complex. The specific compensating transactions depend on the particular systems and processes involved. Implementing Compensating Transactions in Core Java (Simplified Example) # Here\u0026rsquo;s a simplified example in Core Java showcasing the essential aspects of the Compensating Transaction Pattern:\nScenario : Booking a hotel room with payment.\nClasses :\nHotelBookingService: Maintains hotel booking logic. PaymentService: Handles payment processing. Booking: Represents a hotel room booking with details. Code:\npublic class HotelBookingService { private final PaymentService paymentService; public HotelBookingService(PaymentService paymentService) { this.paymentService = paymentService; } public Booking bookRoom(String guestName, String roomType, double price) { // 1. Book the room (potentially in a database) Booking booking = new Booking(guestName, roomType, price); // ... (simulate database booking) // 2. Process payment boolean paymentSuccess = paymentService.charge(price); if (paymentSuccess) { return booking; // success } else { // 3. Compensate if payment fails: cancelBooking(booking); throw new RuntimeException(\u0026#34;Payment failed, booking cancelled.\u0026#34;); } } private void cancelBooking(Booking booking) { // ... (simulated database cancellation) System.out.println(\u0026#34;Booking cancelled for room: \u0026#34; + booking.getRoomType()); } } public class PaymentService { public boolean charge(double amount) { // ... (simulated payment processing - can throw exceptions) System.out.println(\u0026#34;Processing payment of $\u0026#34; + amount); // For simplicity, always succeed in this example return true; } } public class Booking { private String guestName; private String roomType; private double price; // getters, setters, and constructors omitted for brevity } Explanation :\n“HotelBookingService.bookRoom ” initiates the main transaction: * Books the room (simulated with a comment). * Processes payment using PaymentService. If payment succeeds, the booked room is returned. If payment fails : * \u0026#34;**cancelBooking** \u0026#34; is called to undo the room booking (simulated). * A \u0026#34;**RuntimeException** \u0026#34; is thrown to indicate failure. Note :\nThis is a simplified example for learning purposes and doesn\u0026rsquo;t handle real-world complexities like concurrency, resource management, error handling, and persistence. Payment service is always successful here for simplicity. In reality, it might fail, requiring more elaborate compensation logic. This is just a single example to illustrate the concepts. Actual implementations vary based on specific scenarios and technology stacks.\nWhat other Patterns are often combined? # The Compensating Transaction Pattern is often combined with other design patterns to create robust and flexible solutions for handling complex transactions and distributed systems. Some patterns that are frequently combined with the Compensating Transaction Pattern include:\nRetry Pattern: # This pattern involves retrying an failed operation, hoping it will succeed on subsequent attempts. Combining this pattern with compensating transactions can improve fault tolerance by automatically retrying the failed operation before executing compensating actions.\nSaga Pattern: # The Saga Pattern decomposes a long-running transaction into a series of smaller, localised transactions. Each step in the saga is executed atomically, and compensating transactions are defined to undo the effects of completed steps if a failure occurs. This pattern is highly compatible with compensating transactions and helps manage complex distributed transactions.\nEvent Sourcing Pattern: # In event sourcing, the state of an application is determined by a sequence of events. This pattern can be combined with compensating transactions to ensure that events representing successful transactions are persisted only after the compensating actions for the entire transaction have been successfully executed.\nCircuit Breaker Pattern: # The Circuit Breaker Pattern detects and prevents repeated failures when calling remote services. When combined with compensating transactions, the circuit breaker can temporarily stop sending requests to a failing service, allowing the system to execute compensating actions and recover from the failure.\nIdempotent Receiver Pattern: # This pattern ensures receiving systems can safely process the same message multiple times without causing unintended side effects. When combined with compensating transactions, idempotency guarantees that executing compensating actions numerous times does not lead to inconsistencies in the system.\nBy combining these patterns, developers can design resilient and scalable distributed systems that can effectively handle failures, maintain consistency, and recover from errors gracefully.\nConclusion: # In conclusion, the Compensating Transaction Pattern offers valuable benefits such as fault tolerance, data consistency, and resilience in distributed systems by providing a mechanism to handle transaction failures and maintain system integrity. However, leveraging this pattern effectively requires careful consideration of its tradeoffs and complexities.\nWhile compensating transactions enhance fault tolerance and data consistency, they introduce development and testing complexity due to the intricacies of designing and validating compensating actions and ensuring the correctness and robustness of transactional workflows. Challenges such as concurrency, error handling, integration, and end-to-end testing require thorough planning, design, and validation to mitigate.\nDespite these challenges, the Compensating Transaction Pattern remains a powerful tool for building resilient and scalable distributed systems. By addressing the complexities and tradeoffs associated with this pattern through effective design, testing, and validation strategies, developers can harness its benefits to build reliable and fault-tolerant systems that meet the demands of modern distributed computing environments.\n","date":"12 February 2024","externalUrl":null,"permalink":"/posts/the-compensating-transaction-pattern/","section":"Posts","summary":"The Bird-Eye View # A Compensating Transaction Pattern is a technique used to ensure consistency when multiple steps are involved in a process, and some steps may fail. It essentially consists in having “undo” transactions for each successful step, so if something goes wrong later on, you can reverse the changes made earlier and maintain data integrity.\n","title":"The Compensating Transaction Pattern","type":"posts"},{"content":"Serialisation in Java is implemented to convert the state of an object into a byte stream, which can be quickly persisted to a file or sent over a network. This process is essential for persisting object data, supporting network communication, and facilitating sharing of objects between different parts of a distributed system.\nBasics of Serialization: # Object Serialization: # - Serialization is converting an object into a sequence of bytes.\n- The serialised form of an object can be saved to a file or sent over a network.\n- Deserialization is the process of reconstructing the object from its serialised form.\nSerializable Interface: # - In Java, the java.io.Serializable interface makes a class serialisable.\n- This interface acts as a marker interface, indicating that class objects can be serialised.\nTransient Keyword: # - The transient keyword can mark instance variables that should not be serialised.\n- Transient variables are not included in the serialised form when an object is serialised.\nclass MyClass implements Serializable { int normalVar; transient int transientVar; } The Serialisation Process: # ObjectOutput/InputStream: # - The ObjectOutputStream class is used for serialising objects.\n- The ObjectInputStream class is used for deserialising objects.\nSerialisable Fields: # - Only the serialisation process will include the fields of a class marked as serialisable (i.e., non-transient and their types are serialisable).\nSerialisable Methods: # - If a class provides its own writeObject and readObject methods, these methods will be responsible for the custom serialisation and deserialisation logic.\nprivate void writeObject(ObjectOutputStream out) throws IOException { // Custom serialisation logic } private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { // Custom deserialisation logic } Serialisation Control: # SerialVersionUID: # - Every serialisable class has a version number, known as serialVersionUID.\n- It is used during deserialisation to verify that the sender and receiver of a serialised object have loaded classes for that object that are compatible concerning serialisation.\nExternalisable Interface: # In Java, the **Externalizable** interface provides a way to customise an object\u0026rsquo;s serialisation and deserialisation process. Unlike the Serializable interface, which performs automatic serialisation, **Externalizable** allows you more control over the process by implementing two methods: **writeExternal** and **readExternal** .\nHere\u0026rsquo;s a simple example to demonstrate the use of Externalizable:\nimport java.io.*; class Person implements Externalizable { private String name; private int age; // Required public no-argument constructor public Person() { } public Person(String name, int age) { this.name = name; this.age = age; } @Override public void writeExternal(ObjectOutput out) throws IOException { // Custom serialization logic out.writeObject(name); out.writeInt(age); } @Override public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException { // Custom deserialization logic name = (String) in.readObject(); age = in.readInt(); } @Override public String toString() { return \u0026#34;Person{name=\u0026#39;\u0026#34; + name + \u0026#34;\u0026#39;, age=\u0026#34; + age + \u0026#34;}\u0026#34;; } } public class ExternalizableExample { public static void main(String[] args) { // Create a Person object Person person = new Person(\u0026#34;John\u0026#34;, 30); // Serialize the object to a file try (ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(\u0026#34;person.ser\u0026#34;))) { oos.writeObject(person); } catch (IOException e) { e.printStackTrace(); } // Deserialize the object from the file Person deserializedPerson = null; try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(\u0026#34;person.ser\u0026#34;))) { deserializedPerson = (Person) ois.readObject(); } catch (IOException | ClassNotFoundException e) { e.printStackTrace(); } // Print the deserialized object if (deserializedPerson != null) { System.out.println(\u0026#34;Deserialized Person: \u0026#34; + deserializedPerson); } } } In this example, the Person class implements the Externalizable interface and provides custom serialisation and deserialisation logic in the **writeExternal** and **readExternal** methods. The main method demonstrates how to create a Person object, serialise it to a file, and then deserialise it back into a new object.\nWhy should I use Externalizable? # The **Externalizable** interface in Java provides a more flexible and customised approach to serialisation compared to the default serialisation mechanism provided by the Serializable interface. Here are some reasons you might want to use Externalizable:\nSelective Serialization : # With **Externalizable** , you have complete control over which fields get serialised and how they are serialised. This can be useful when you exclude certain fields from serialisation or perform custom serialisation for specific fields.\nPerformance Optimization: # **Externalizable** allows you to implement custom serialisation logic that might be more efficient than the default serialisation mechanism. You can choose to serialise only essential data, avoiding unnecessary information and potentially reducing the size of the serialised data.\nVersioning Control: # When your class evolves and you need to maintain backwards or forward compatibility, Externalizable provides a way to handle versioning explicitly. You can implement custom logic to manage different versions of the serialised data, making it more adaptable to changes in your class structure.\nSecurity Considerations: # Custom serialisation logic can help handle security-related concerns. For example, encrypt sensitive information before serialisation or perform other security checks during the serialisation/deserialisation process.\nResource Management: # With **Externalizable** , you control resource management during serialisation and deserialisation. You can explicitly release and acquire resources as needed, which can be crucial when resources must be managed carefully.\nReducing Serialized Size: # If your class has transient or derived fields that need not be serialised, you can exclude them from the serialised form, leading to a smaller serialised size.\nIt\u0026rsquo;s worth noting that while **Externalizable** provides more control, it also requires more manual effort to implement, as you need to write custom serialisation and deserialisation logic. In many cases, the default serialisation provided by Serializable is sufficient and more straightforward. However, **Externalizable** offers a powerful alternative for scenarios where customisation is necessary.\nSummary : # Serialisation is a fundamental concept in Java that facilitates the storage and transmission of object data. It provides a versatile mechanism for persisting object states, supporting network communication, and enabling interoperability between distributed system components. While the default serialisation mechanism is suitable for many cases, understanding advanced topics like custom serialisation, versioning, and security considerations is essential for developing robust and efficient Java applications.\nUseCases for Serialisation in Java # The main reasons for implementing serialisation in Java are as follows:\nPersistence : # Serialisation allows objects to be saved to a file and later restored. This is useful for applications that need to store and retrieve the state of objects, such as in database interactions or for caching purposes.\nPersistence - Example: # Let\u0026rsquo;s consider a simple example where we have a Person class that we want to persist to a file using serialisation.\nimport java.io.*; class Person implements Serializable { private String name; private int age; public Person(String name, int age) { this.name = name; this.age = age; } @Override public String toString() { return \u0026#34;Person{name=\u0026#39;\u0026#34; + name + \u0026#34;\u0026#39;, age=\u0026#34; + age + \u0026#39;}\u0026#39;; } } public class SerializationExample { public static void main(String[] args) { // Create a Person object Person person = new Person(\u0026#34;John Doe\u0026#34;, 25); // Serialize the object to a file serializePerson(person, \u0026#34;person.ser\u0026#34;); // Deserialize the object from the file Person deserializedPerson = deserializePerson(\u0026#34;person.ser\u0026#34;); // Display the deserialised object System.out.println(\u0026#34;Deserialized Person: \u0026#34; + deserializedPerson); } private static void serializePerson(Person person, String fileName) { try (ObjectOutputStream outputStream = new ObjectOutputStream(new FileOutputStream(fileName))) { // Write the object to the file outputStream.writeObject(person); System.out.println(\u0026#34;Serialization successful. Object saved to \u0026#34; + fileName); } catch (IOException e) { e.printStackTrace(); } } private static Person deserializePerson(String fileName) { Person person = null; try (ObjectInputStream inputStream = new ObjectInputStream(new FileInputStream(fileName))) { // Read the object from the file person = (Person) inputStream.readObject(); System.out.println(\u0026#34;Deserialization successful. Object loaded from \u0026#34; + fileName); } catch (IOException | ClassNotFoundException e) { e.printStackTrace(); } return person; } } In this example, the Person class implements the Serializable interface, indicating that its instances can be serialised. The SerializationExample class demonstrates how to serialise a Person object to a file (person.ser) and then deserialise it back into a new Person object. The serializePerson and deserializePerson methods handle the serialisation and deserialisation processes, respectively.\nNetwork Communication : # Serialisation enables the easy transmission of objects between different Java Virtual Machines (JVMs) over a network. Converting objects to a byte stream can be sent across a network and reconstructed on the receiving end.\nNetwork Communication - Example: # In this example, we\u0026rsquo;ll create a simple client-server communication scenario using Java sockets and serialisation. The server will send a serialised object to the client over the network.\nLet\u0026rsquo;s start with the server:\nimport java.io.*; import java.net.ServerSocket; import java.net.Socket; class Person implements Serializable { private String name; private int age; public Person(String name, int age) { this.name = name; this.age = age; } @Override public String toString() { return \u0026#34;Person{name=\u0026#39;\u0026#34; + name + \u0026#34;\u0026#39;, age=\u0026#34; + age + \u0026#39;}\u0026#39;; } } public class Server { public static void main(String[] args) { try { // Create a server socket ServerSocket serverSocket = new ServerSocket(12345); System.out.println(\u0026#34;Server listening on port 12345...\u0026#34;); while (true) { // Wait for a client to connect Socket clientSocket = serverSocket.accept(); System.out.println(\u0026#34;Client connected: \u0026#34; + clientSocket.getInetAddress()); // Create a Person object Person person = new Person(\u0026#34;Alice\u0026#34;, 30); // Serialize and send the object to the client sendObjectToClient(person, clientSocket); // Close the client socket clientSocket.close(); } } catch (IOException e) { e.printStackTrace(); } } private static void sendObjectToClient(Serializable obj, Socket socket) { try (ObjectOutputStream outputStream = new ObjectOutputStream(socket.getOutputStream())) { // Write the object to the output stream outputStream.writeObject(obj); System.out.println(\u0026#34;Object sent to client.\u0026#34;); } catch (IOException e) { e.printStackTrace(); } } } And here\u0026rsquo;s the client:\nimport java.io.*; import java.net.Socket; public class Client { public static void main(String[] args) { try { // Connect to the server Socket socket = new Socket(\u0026#34;localhost\u0026#34;, 12345); // Receive the serialised object from the server Person receivedPerson = receiveObjectFromServer(socket); // Display the received object System.out.println(\u0026#34;Received Person from server: \u0026#34; + receivedPerson); // Close the socket socket.close(); } catch (IOException | ClassNotFoundException e) { e.printStackTrace(); } } private static Person receiveObjectFromServer(Socket socket) throws IOException, ClassNotFoundException { try (ObjectInputStream inputStream = new ObjectInputStream(socket.getInputStream())) { // Read the object from the input stream return (Person) inputStream.readObject(); } } } In this example, the Server class listens for incoming connections on port 12345. When a client connects, it creates a Person object, serialises it, and sends it to the client using the sendObjectToClient method. The Client class connects to the server, receives the serialised Person object, and then deserialises and prints it.\nObject Cloning : # Serialisation provides a way to create a deep copy of an object. By serialising an object and then deserialising it, you effectively create a new copy of the original object. This can be useful when you need to duplicate complex object structures.\nObject Cloning - Example: # In this example, I\u0026rsquo;ll demonstrate object cloning using serialisation in Java. We\u0026rsquo;ll create a Person class and use serialisation and deserialisation to perform deep cloning.\nimport java.io.*; class Address implements Serializable { private String street; private String city; public Address(String street, String city) { this.street = street; this.city = city; } @Override public String toString() { return \u0026#34;Address{street=\u0026#39;\u0026#34; + street + \u0026#34;\u0026#39;, city=\u0026#39;\u0026#34; + city + \u0026#34;\u0026#39;}\u0026#34;; } } class Person implements Serializable { private String name; private int age; private Address address; public Person(String name, int age, Address address) { this.name = name; this.age = age; this.address = address; } @Override public String toString() { return \u0026#34;Person{name=\u0026#39;\u0026#34; + name + \u0026#34;\u0026#39;, age=\u0026#34; + age + \u0026#34;, address=\u0026#34; + address + \u0026#39;}\u0026#39;; } } public class ObjectCloningExample { public static void main(String[] args) { // Create a Person object Address originalAddress = new Address(\u0026#34;123 Main St\u0026#34;, \u0026#34;Cityville\u0026#34;); Person originalPerson = new Person(\u0026#34;John Doe\u0026#34;, 25, originalAddress); // Clone the object using serialisation and deserialisation Person clonedPerson = cloneObject(originalPerson); // Modify the original object originalPerson.setName(\u0026#34;Jane Doe\u0026#34;); originalPerson.setAge(30); originalAddress.setStreet(\u0026#34;456 Oak St\u0026#34;); // Display the original and cloned objects System.out.println(\u0026#34;Original Person: \u0026#34; + originalPerson); System.out.println(\u0026#34;Cloned Person: \u0026#34; + clonedPerson); } private static Person cloneObject(Person original) { try { // Serialize the original object ByteArrayOutputStream bos = new ByteArrayOutputStream(); ObjectOutputStream out = new ObjectOutputStream(bos); out.writeObject(original); out.flush(); // Deserialize the object to create a deep copy ByteArrayInputStream bis = new ByteArrayInputStream(bos.toByteArray()); ObjectInputStream in = new ObjectInputStream(bis); return (Person) in.readObject(); } catch (IOException | ClassNotFoundException e) { e.printStackTrace(); return null; } } } The Person class contains an Address object in this example. The ObjectCloningExample class demonstrates how to clone a Person object by serialising it to a byte stream and then deserialising it to create a deep copy. The cloneObject method handles the serialisation and deserialisation process. After cloning, modifications to the original object do not affect the cloned object, demonstrating a deep copy.\nRemote Method Invocation (RMI) : # Java Remote Method Invocation (RMI) is a mechanism that allows objects to invoke methods on objects in another JVM. Serialisation is used to marshal and unmarshal parameters and return values during remote method calls.\nJavaBeans : # Serialisation is commonly used in JavaBeans, reusable software components for Java. JavaBeans often need to be serialised to be easily stored or transmitted.\nHeads Up # Despite these advantages, it\u0026rsquo;s important to note that not all classes can or should be serialised. For a class to be serialisable, it must implement the Serializable interface. Additionally, care should be taken when serialising objects to ensure security and proper handling of changes to class definitions over time (versioning).\nWhat are the security issues? # Java Serialization can introduce security issues, particularly when deserialising data from untrusted sources. Here are some of the security concerns associated with Java Serialization:\nRemote Code Execution: # One of the most significant security risks with Java Serialization is the potential for remote code execution. When deserialising an object, the Java runtime system can execute arbitrary code within the serialised data. Attackers can exploit this to execute malicious code on the target system. This vulnerability can lead to serious security breaches.\nDenial of Service (DoS): # An attacker can create a serialised object with a large size, causing excessive memory consumption and potentially leading to a denial of service attack. Deserialising large objects can consume significant CPU and memory resources, slowing down or crashing the application.\nData Tampering: # Serialised data can be tampered with during transmission or storage. Attackers can modify the serialised byte stream to alter the state of the deserialised object or introduce vulnerabilities.\nInsecure Deserialization: # Deserialising untrusted data without proper validation can lead to security issues. For example, if a class that performs sensitive operations is deserialised from untrusted input, an attacker can manipulate the object\u0026rsquo;s state to perform unauthorised actions.\nInformation Disclosure: # When objects are serialised, sensitive information may be included in the serialised form. An attacker may gain access to sensitive information if this data needs to be adequately protected or encrypted.\nHow to Mitigate Serialization Issues # To mitigate these security issues, consider the following best practices:\nAvoid Deserializing Untrusted Data: # Avoid deserialising data from untrusted sources altogether. Instead, use safer data interchange formats like JSON or XML for untrusted data.\nImplement Input Validation: # When deserialising data, validate and sanitise the input to ensure it adheres to expected data structures and doesn\u0026rsquo;t contain unexpected or malicious data.\nUse Security Managers: # Java\u0026rsquo;s Security Manager can be used to restrict the permissions and actions of deserialised code. However, it\u0026rsquo;s important to note that security managers have been removed in newer versions of Java.\nWhitelist Classes: # Limit the classes that can be deserialised to a predefined set of trusted classes. This can help prevent the deserialisation of arbitrary and potentially malicious classes.\nVersioning and Compatibility: # Be cautious when making changes to serialised classes. Use serialVersionUID to manage versioning and compatibility between different versions of serialised objects.\nWhat is serialVersionUID in Java, and how does it work? # In Java, the serialVersionUID is a unique identifier associated with a serialisable class. It is used during object serialisation to verify that the sender and receiver of a serialised object have loaded classes for that object that are compatible with serialisation. If the receiver has loaded a class for the object with a different serialVersionUID than the corresponding class on the sender side, deserialisation will result in an InvalidClassException.\nHere\u0026rsquo;s how it works:\nAutomatic Serialization: # When you mark a class as Serializable in Java, the serialisation mechanism automatically generates a serialVersionUID for that class unless you provide one explicitly. This autogenerated serialVersionUID is based on various aspects of the class, including its name, implemented interfaces, fields, and methods.\nimport java.io.Serializable; public class MyClass implements Serializable { // class code } Explicit serialVersionUID: # You can also explicitly declare a serialVersionUID in your class to have more control over versioning. This can be useful to avoid unexpected serialisation compatibility issues when the class structure changes.\nimport java.io.Serializable; public class MyClass implements Serializable { private static final long serialVersionUID = 123456789L; // class code } It\u0026rsquo;s important to note that if you don\u0026rsquo;t provide an explicit serialVersionUID, the Java runtime will automatically generate one based on the class\u0026rsquo;s structure, and changes to the class may result in a different autogenerated ID.\nVersioning: # When an object is serialised, the serialVersionUID is also stored in the serialised form. During deserialisation, the receiving end compares the serialVersionUID of the loaded class with the one stored in the serialised data. If they match, deserialisation proceeds; otherwise, an InvalidClassException is thrown.\nThis mechanism helps ensure that the serialised data is compatible with the current class version. If you make changes to the class that are not backwards-compatible, you should update the serialVersionUID to indicate that the new class is incompatible with the previous version.\nIn summary, serialVersionUID in Java is a version control mechanism for serialised objects, and it plays a crucial role in maintaining compatibility between different versions of a class during object serialisation and deserialisation.\nSerialVersionUID Example # Imagine you have a Person class that represents a person with a name and an age, and you want to serialise instances of this class.\nimport java.io.*; public class Person implements Serializable { private static final long serialVersionUID = 1L; private String name; private int age; public Person(String name, int age) { this.name = name; this.age = age; } // getters and setters @Override public String toString() { return \u0026#34;Person{name=\u0026#39;\u0026#34; + name + \u0026#34;\u0026#39;, age=\u0026#34; + age + \u0026#39;}\u0026#39;; } public static void main(String[] args) { // Serialization serializePerson(); // Deserialization Person deserializedPerson = deserializePerson(); System.out.println(\u0026#34;Deserialized Person: \u0026#34; + deserializedPerson); } private static void serializePerson() { try (ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(\u0026#34;person.ser\u0026#34;))) { Person person = new Person(\u0026#34;John\u0026#34;, 25); oos.writeObject(person); System.out.println(\u0026#34;Person serialized successfully.\u0026#34;); } catch (IOException e) { e.printStackTrace(); } } private static Person deserializePerson() { try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(\u0026#34;person.ser\u0026#34;))) { return (Person) ois.readObject(); } catch (IOException | ClassNotFoundException e) { e.printStackTrace(); return null; } } } In this example:\n- The Person class implements the Serializable interface.\n- It has a serialVersionUID field explicitly set to 1L.\n- The serializePerson method creates a Person object, serialises it, and writes it to a file called person.ser.\n- The deserializePerson method reads and returns the serialised Person object from the file.\nNow, let\u0026rsquo;s see how versioning works:\n1. Run the program to serialise a Person object and write it to the file.\n2. Change the Person class by adding a new field, for example, private boolean isStudent;.\n3. rerun the program to deserialise the Person object.\nIf you don\u0026rsquo;t update the serialVersionUID when you add the new field, you will likely encounter an InvalidClassException during deserialisation. To avoid this, you should update the serialVersionUID to indicate that the new version of the class is not compatible with the previous one:\n**private static final long serialVersionUID = 2L;** When you rerun the program, it should deserialise the object successfully, and the isStudent field will have its default value (false). Remember that this is a simple example; in a real-world scenario, you might need to implement more sophisticated versioning strategies based on your application\u0026rsquo;s requirements.\nSecurity Libraries: # Consider using third-party libraries like Apache Commons Collections or OWASP Java Serialization Security (Java-Serial-Killer) to help mitigate known vulnerabilities and prevent common attacks.\nLessons Learned # In summary, Java Serialization can introduce serious security risks, especially when dealing with untrusted data. It\u0026rsquo;s essential to take precautions, validate inputs, and consider alternative serialisation methods or libraries to enhance security. Additionally, keeping your Java runtime environment up to date is crucial, as newer versions of Java may include security improvements and fixes for known vulnerabilities.\nConclusion: # Overall, serialisation is both a powerful and dangerous tool. We must balance usage and comfort at each location against our security needs. If you look at open-source projects, you can see a lot of activity to minimise the risk and increase the abstraction of this topic for everyday use.\n","date":"11 February 2024","externalUrl":null,"permalink":"/posts/serialising-in-java-birds-eye-view/","section":"Posts","summary":"Serialisation in Java is implemented to convert the state of an object into a byte stream, which can be quickly persisted to a file or sent over a network. This process is essential for persisting object data, supporting network communication, and facilitating sharing of objects between different parts of a distributed system.\n","title":"Serialising in Java - Birds Eye View","type":"posts"},{"content":"","date":"8 February 2024","externalUrl":null,"permalink":"/tags/coordinate-system/","section":"Tags","summary":"","title":"Coordinate System","type":"tags"},{"content":"","date":"8 February 2024","externalUrl":null,"permalink":"/tags/mgrs/","section":"Tags","summary":"","title":"MGRS","type":"tags"},{"content":"","date":"8 February 2024","externalUrl":null,"permalink":"/tags/utm/","section":"Tags","summary":"","title":"UTM","type":"tags"},{"content":"","date":"8 February 2024","externalUrl":null,"permalink":"/tags/utmref/","section":"Tags","summary":"","title":"UTMref","type":"tags"},{"content":"","date":"8 February 2024","externalUrl":null,"permalink":"/tags/wgs84/","section":"Tags","summary":"","title":"WGS84","type":"tags"},{"content":"UTMref stands for Universal Transverse Mercator (UTM) reference system. It is a coordinate system to locate positions on the Earth\u0026rsquo;s surface. The UTM system divides the Earth into a series of zones, each 6 degrees of longitude wide, and assigns a coordinate grid to each zone. This grid system uses easting (measured in meters east of a reference meridian) and northing (measured in meters north of the equator) coordinates to define locations within each zone.\nThe UTMref system provides a convenient way to specify precise locations on the Earth\u0026rsquo;s surface for various purposes, including mapping, navigation, and surveying. It is commonly used in applications such as GPS devices, geographic information systems (GIS), and topographic maps.\nAn UTMref Example: What Zones are covered by Germany? Who is using UTMref? Surveying and Mapping: Navigation: Geographic Information Systems (GIS): Engineering and Construction: Natural Resource Management: Emergency Response and Disaster Management: Military Applications: Components of MGRS: Example of MGRS Coordinate: History: Scientific Research: What is the relationship between UTMref and UTM? What is the difference between UTMref and MGRS? https://youtu.be/P2sUz-Vj-mo\nAn UTMref Example: # Here\u0026rsquo;s an example of a location specified using the UTMref coordinate system:\nLatitude: 40.7128° N\nLongitude: -74.0060° W\nUsing a UTM reference system, this location could be expressed in terms of easting and northing coordinates within a specific UTM zone. Let\u0026rsquo;s assume this location falls within UTM zone 18T, which covers part of the eastern United States.\nUTM Zone: 18T\nEasting: 583,784 meters\nNorthing: 4,501,180 meters\nThese easting and northing coordinates represent the position of the location within the designated UTM zone, providing a precise way to reference the location on the Earth\u0026rsquo;s surface.\nWhat Zones are covered by Germany? # Several UTM zones cover Germany, as they span multiple longitudinal extents. The UTM zones covering Germany are primarily 32N, 33N, and 32U.\nUTM Zone 32N covers the westernmost part of Germany, including cities like Aachen, Cologne and parts of the Rhineland.\nUTM Zone 33N covers central Germany, including cities like Frankfurt, Stuttgart, Munich, and much of Bavaria.\nUTM Zone 32U covers the easternmost part of Germany, including cities like Berlin, Leipzig, Dresden, and parts of Brandenburg and Saxony.\nThese zones ensure accurate representation and measurement of locations within Germany using the UTM coordinate system.\nWho is using UTMref? # The UTMref (Universal Transverse Mercator reference system) is commonly used in various fields and applications where precise location referencing is required. Some of the key users of UTMref include:\nSurveying and Mapping: # UTMref coordinates are widely used in surveying, cartography, and mapping applications to represent geographic features, boundaries, and infrastructure accurately.\nNavigation: # UTMref coordinates are used in GPS devices, navigation systems, and aviation for determining and communicating precise locations.\nGeographic Information Systems (GIS): # UTMref coordinates are used extensively in GIS software for spatial data analysis, visualization, and mapping.\nEngineering and Construction: # UTMref coordinates are used in engineering projects, construction planning, and infrastructure development to locate project sites, structures, and utilities precisely.\nNatural Resource Management: # UTMref coordinates are used in forestry, agriculture, land management, and environmental monitoring to inventory resources, assess land use, and manage natural habitats.\nEmergency Response and Disaster Management: # UTMref coordinates are used by emergency responders, search and rescue teams, and disaster management agencies for coordinating efforts, locating incidents, and navigating affected areas.\nMilitary Applications: # UTMref coordinates, particularly the Military Grid Reference System (MGRS), are extensively used in military operations for precise location referencing, target designation, and navigation. It is based on the Universal Transverse Mercator (UTM) system but adds additional elements to facilitate precise location referencing. MGRS provides a concise and consistent method for specifying any point on the Earth\u0026rsquo;s surface.\nComponents of MGRS: # Grid Zone Designator (GZD) : The first element of an MGRS coordinate specifies the UTM grid zone in which the location falls. It consists of a letter representing one of the 6-degree longitudinal zones (A through Z, excluding I) and a letter designating one of 20 latitude bands (C through X, excluding I and O).\n100,000-meter Grid Square Identifier : The second element of an MGRS coordinate identifies a 100,000-meter square within the specified UTM grid zone. A pair of letters designate it.\nNumerical Coordinates within Grid Square : The third element of an MGRS coordinate represents the numerical coordinates (easting and northing) within the 100,000-meter grid square. These coordinates are expressed as digits, typically ranging from 0 to 99,999.\nExample of MGRS Coordinate: # Let\u0026rsquo;s consider an example MGRS coordinate: 33TWN1234567890.\n- Grid Zone Designator (GZD): 33T\n- 100,000-meter Grid Square Identifier: WN\n- Numerical Coordinates within Grid Square: 1234567890\nHistory: # Origin : MGRS was developed by the United States Army as a military standard for specifying locations on the Earth\u0026rsquo;s surface. Its development was influenced by the need for a standardized, easy-to-use coordinate system for military operations, especially in diverse geographic regions and adverse conditions.\nStandardization : MGRS was standardized by the North Atlantic Treaty Organization (NATO) in the mid-20th century to ensure interoperability among military forces of member countries. This standardization facilitated communication, coordination, and navigation across allied military units.\nUsage : MGRS has been extensively used in military operations, including land navigation, target designation, logistics planning, and communication of precise locations between units. Its adoption has expanded to civilian applications, including emergency response, search and rescue, GIS, and outdoor recreation.\nMGRS remains a critical component of military operations and continues to be employed by armed forces worldwide due to its accuracy, simplicity, and interoperability. Its standardized format and global coverage make it valuable for diverse military and civilian applications requiring precise location referencing.\nScientific Research: # UTMref coordinates are used in various scientific disciplines, such as geology, ecology, archaeology, and climatology, for fieldwork, data collection, and spatial analysis.\nOverall, UTMref provides a standardized and efficient way to specify locations on the Earth\u0026rsquo;s surface, making it a valuable tool across various industries and disciplines.\nWhat is the relationship between UTMref and UTM? # UTMref (Universal Transverse Mercator reference system) and UTM (Universal Transverse Mercator) are closely related concepts, with UTMref being a specific implementation or usage of the UTM coordinate system.\nUTM (Universal Transverse Mercator) : UTM is a global coordinate system used to specify locations on the Earth\u0026rsquo;s surface. It divides the Earth into multiple zones, each 6 degrees of longitude wide, and employs a transverse Mercator projection to map locations within each zone. UTM coordinates consist of easting and northing values (measured in meters) relative to a reference point within each zone.\nUTMref (Universal Transverse Mercator reference system) : UTMref is the practical usage or application of the UTM coordinate system. It involves assigning specific UTM coordinates to reference points or locations on the Earth\u0026rsquo;s surface. UTMref coordinates are commonly used in various fields such as surveying, mapping, navigation, GIS, and others for accurately representing and referencing locations.\nIn summary, UTM is the coordinate system, while UTMref refers to using UTM coordinates to reference specific locations. UTMref is a practical implementation of the UTM system in various applications and industries where precise location referencing is required.\nWhat is the difference between UTMref and MGRS? # UTMref (Universal Transverse Mercator reference system) and MGRS (Military Grid Reference System) are closely related coordinate systems commonly used for identifying locations on the Earth\u0026rsquo;s surface. While they share similarities, there are also differences between them:\nGrid System : UTMref and MGRS are based on the UTM projection, which divides the Earth into zones and uses a rectangular grid system. However, MGRS adds refinement by dividing each UTM zone into a further grid of 100,000-meter squares called grid zones.\nCoordinate Format : UTMref coordinates are typically expressed in terms of easting (distance east of a reference meridian) and northing (distance north of the equator) within a specific UTM zone. For example, in UTMref, a coordinate might be \u0026ldquo;32U 123456 9876543\u0026rdquo;, where \u0026ldquo;32U\u0026rdquo; denotes the UTM zone, and the following numbers represent the easting and northing, respectively.\nMGRS, on the other hand, further divides each UTM zone into a grid of 100,000-meter squares and assigns a letter pair to each square. MGRS coordinates include the UTM zone, the grid square designation, and an additional set of digits representing the easting and northing within that grid square. For example, an MGRS coordinate might look like \u0026ldquo;32U MV 12345 98765\u0026rdquo;, where \u0026ldquo;32U\u0026rdquo; is the UTM zone, \u0026ldquo;MV\u0026rdquo; is the grid square, and the following numbers represent the easting and northing, respectively.\nUsage : While both systems are used for navigation, mapping, and various geographic applications, MGRS is particularly prevalent in military applications due to its standardized format and ease of communication. It provides a concise and accurate way to specify locations on maps and communicate them efficiently in military operations.\nIn summary, while UTMref and MGRS are based on the same underlying UTM projection and are used for similar purposes, MGRS provides a more refined grid system with additional alphanumeric characters to specify locations within each UTM zone, making it especially useful for military and precise navigation applications.\n","date":"8 February 2024","externalUrl":null,"permalink":"/posts/what-is-utmref/","section":"Posts","summary":"UTMref stands for Universal Transverse Mercator (UTM) reference system. It is a coordinate system to locate positions on the Earth’s surface. The UTM system divides the Earth into a series of zones, each 6 degrees of longitude wide, and assigns a coordinate grid to each zone. This grid system uses easting (measured in meters east of a reference meridian) and northing (measured in meters north of the equator) coordinates to define locations within each zone.\n","title":"What is UTMref?","type":"posts"},{"content":"Contextual analysis in cybersecurity involves examining events, actions, or data within the broader context of an organization\u0026rsquo;s IT environment. It is a critical component of a proactive cybersecurity strategy, aiming to understand the significance of activities by considering various factors surrounding them. This multifaceted approach helps cybersecurity professionals identify and respond to potential threats effectively.\nKey Concepts Situational Awareness Normal Scenario: Anomalous Scenario: Situational Awareness Analysis: Potential Explanations: Response: Data Correlation Scenario: Data Sources: Correlation: Correlated Analysis: Response: Threat Intelligence Integration Scenario: Threat Intelligence Integration: Response: User Behavior Analysis Scenario: User Behavior Analysis: Response: Temporal Analysis Scenario: Analysis: Contextual Analysis: Confirmation of Anomaly: Response: Environmental Context Scenario: Environmental Context Analysis: Response Based on Environmental Context: Implementation of Contextual Analysis Security Information and Event Management (SIEM) Systems Machine Learning and Artificial Intelligence Automation and Orchestration Incident Response Planning Challenges and Considerations False Positives and Negatives Data Privacy and Compliance Continuous Monitoring Skill and Resource Requirements Conclusion Key Concepts # Situational Awareness # Contextual analysis begins with establishing situational awareness—a comprehensive understanding of an organization\u0026rsquo;s network, systems, and data. This includes mapping out assets, identifying vulnerabilities, and recognizing standard patterns of behaviour. By understanding the baseline, anomalies and potential threats can be detected more efficiently.\nSituational awareness in cybersecurity can be illustrated through an example involving network activity. Imagine a typical day within an organization where employees access company resources and network traffic follows regular patterns.\nNormal Scenario: # During working hours, employees log in to their computers, access specific servers, and engage in routine activities such as sending emails, accessing shared files, and using approved applications. During this time, the Network logs consistently show a steady flow of data between internal devices.\nAnomalous Scenario: # One day, the network logs indicate an unusually high volume of data being transmitted between an employee\u0026rsquo;s workstation and a server containing sensitive financial information. This activity is occurring during non-working hours when such data transfers are atypical. Simultaneously, the employee\u0026rsquo;s account is attempting to access multiple systems that are not part of their regular job responsibilities.\nSituational Awareness Analysis: # Situational awareness would involve recognizing the deviation from the baseline of regular network activity. Security analysts would investigate the anomalous behaviour, considering factors such as the timing of the activity, the specific systems involved, and the employee\u0026rsquo;s role. By correlating data from various sources like logs, endpoint security tools, and user behaviour analytics, the analysts aim to understand the broader context of the situation.\nPotential Explanations: # It could be a legitimate activity, such as an employee working on an urgent project during non-working hours. However, this should be validated to ensure it\u0026rsquo;s not a compromise or misuse of credentials. Alternatively, it might indicate a compromised account, where an external actor or malicious insider is attempting to exfiltrate sensitive data.\nResponse: # Based on the situational awareness analysis, appropriate responses can be initiated, such as blocking the suspicious activity, notifying the employee to confirm the legitimacy of their actions, and escalating the incident for further investigation if needed. Automated responses, such as temporarily blocking the account, could be triggered to contain potential threats while the situation is assessed.\nIn this example, situational awareness involves understanding the typical patterns of network activity, identifying anomalies, and responding appropriately to potential security threats based on a broader understanding of the context surrounding the events.\nData Correlation # In a cybersecurity context, data is generated from various sources, such as network logs, endpoint devices, and application activity. The contextual analysis involves correlating this diverse set of data to create a more complete picture of an event or potential threat. Correlation helps distinguish between isolated incidents and coordinated attacks.\nLet\u0026rsquo;s consider an example involving a potential security incident and how data correlation helps in identifying and responding to the threat:\nScenario: # An organization has an SIEM (Security Information and Event Management) system in place, collecting data from different sources such as firewall logs, antivirus alerts, and server logs. One day, the antivirus system generates an alert about a suspicious file being detected on a user\u0026rsquo;s workstation.\nData Sources: # Firewall Logs : Show an unusual outbound connection from the same user\u0026rsquo;s workstation to an external IP address.\nAntivirus Alert : Indicates the detection of a malware file on the user\u0026rsquo;s workstation.\nServer Logs : Display an increase in failed login attempts on a critical server from the same user\u0026rsquo;s credentials.\nCorrelation: # Without data correlation, each event might be treated in isolation, potentially leading to incomplete insights into the security incident. Data correlation involves combining information from these different sources to create a more detailed picture of the situation.\nCorrelated Analysis: # Firewall Logs + Antivirus Alert : The outbound connection in the firewall logs and the antivirus alert both relate to the same user\u0026rsquo;s workstation, suggesting that the system might be compromised.\nAntivirus Alert + Server Logs : The antivirus alert indicates a malware file on the user\u0026rsquo;s workstation, and the server logs show an increase in failed login attempts using the same credentials. This correlation suggests that the malware might be attempting to propagate to other systems using stolen credentials.\nFirewall Logs + Server Logs : Combining information from firewall logs and server logs reveals that the suspicious outbound connection coincides with the increase in failed login attempts on the critical server. This correlation strengthens the hypothesis that the compromised workstation is attempting unauthorized access to the server.\nResponse: # With a correlated analysis, security analysts can respond more effectively. They might isolate the compromised workstation from the network, initiate a malware scan, reset the affected user\u0026rsquo;s credentials, and investigate the attempted unauthorized access to the critical server.\nIn this example, data correlation helps connect the dots between seemingly isolated security events, enabling a more accurate understanding of the incident. It allows security teams to respond promptly and comprehensively to potential threats by considering the broader context of the data.\nThreat Intelligence Integration # Contextual analysis benefits from the integration of threat intelligence. By leveraging information about known threats, vulnerabilities, and attack techniques, organizations can contextualize their security data. This integration enhances the ability to identify and respond to emerging threats that align with known patterns.\nHere\u0026rsquo;s an example illustrating how threat intelligence integration can be applied:\nScenario: # An organization operates in the financial sector and has a robust cybersecurity infrastructure, including an SIEM system and an intrusion detection system (IDS). The organization subscribes to a threat intelligence feed that provides real-time information about emerging cyber threats, including indicators of compromise (IoCs), malware signatures, and tactics, techniques, and procedures (TTPs) associated with specific threat actors.\nThreat Intelligence Integration: # New Threat Indicator Detected : The threat intelligence feed identifies a new malicious IP address associated with a known cybercriminal group targeting financial institutions.\nIntegration with SIEM : The threat intelligence feed is integrated into the organization\u0026rsquo;s SIEM system. This integration allows the SIEM to update its rules and correlation logic automatically based on the newly identified threat indicator.\nSIEM Alerts Triggered : As the SIEM continuously monitors network and system activities, it detects network traffic attempting to communicate with the identified malicious IP address. This triggers alerts within the SIEM system.\nCorrelation with IDS Alerts : Simultaneously, the IDS generates alerts about suspicious activities that align with the TTPs associated with the cybercriminal group mentioned in the threat intelligence feed.\nAnalysis and Confirmation : Security analysts, equipped with the integrated threat intelligence, can quickly correlate the SIEM and IDS alerts. They recognize that the detected activities match the known patterns of the cybercriminal group targeting financial institutions.\nResponse: # With the contextual information from the threat intelligence feed, the security team can tailor their response more effectively. They may block the malicious IP address at the firewall, update antivirus signatures, and implement additional security controls to mitigate the specific threats associated with the identified threat actor.\nIn this example, threat intelligence integration allows the organization to leverage external insights to fortify its defences and respond swiftly to specific threats. This collaborative approach enhances the organization\u0026rsquo;s cybersecurity resilience against targeted attacks.\nUser Behavior Analysis # Understanding typical user behaviour is crucial for contextual analysis. This involves recognizing patterns of activity associated with legitimate users and identifying anomalies that may indicate compromised accounts or malicious insider activities. User behaviour analytics tools play a significant role in this aspect of contextual analysis.\nHere\u0026rsquo;s an example illustrating how user behaviour analysis can be applied:\nScenario: # A medium-sized technology company has implemented user behaviour analysis as part of its cybersecurity strategy. The company\u0026rsquo;s network includes various departments, each with specific access rights to sensitive data and systems. David, a software developer, typically accesses a particular set of servers and repositories related to his development tasks.\nUser Behavior Analysis: # Baseline Establishment : The system establishes a baseline of normal behaviour for David. This includes the typical servers he accesses, the times he logs in, and the types of files he interacts with.\nAnomaly Detection : One day, the user behaviour analysis system flags an anomaly. David is accessing a server he has never accessed before, and he\u0026rsquo;s doing so during non-working hours.\nCorrelation with Other Data : The system correlates this anomaly with other data sources, such as firewall logs and VPN activity. It discovers that, around the same time, there is an unusual outbound connection from David\u0026rsquo;s workstation to an external IP address.\nContextual Analysis : Security analysts perform contextual analysis. They check the nature of the server David accessed and the destination of the outbound connection. They also cross-reference this with threat intelligence to see if the external IP address is associated with known malicious activity.\nConfirmation of Anomaly : Through the analysis, it is confirmed that David\u0026rsquo;s account has been compromised. An attacker gained access to his credentials, leading to unauthorized access to a server and an attempt to communicate with a potentially malicious external entity.\nResponse: # The security team takes immediate action. They isolate John\u0026rsquo;s compromised account, reset his credentials, and investigate the extent of the compromise. Simultaneously, they implement additional security measures, such as two-factor authentication, to prevent future unauthorized access.\nIn this example, user behaviour analysis assists in identifying a potential security incident involving a compromised user account. By continuously monitoring and analyzing user activities, organizations can enhance their ability to detect and respond to security threats in a timely manner.\nTemporal Analysis # Contextual analysis considers the timing of events. Analyzing when certain activities occur can provide insights into their legitimacy. For example, a user accessing sensitive data during non-working hours might raise suspicions. Temporal analysis helps in distinguishing between normal and potentially malicious activities.\nHere\u0026rsquo;s an example illustrating how temporal analysis can be applied:\nScenario: # A financial institution operates a web-based application that facilitates online transactions for its customers. The system logs various events, including user logins, financial transactions, and database access.\nAnalysis: # Baseline Establishment : The security team establishes a baseline of standard temporal patterns for user logins and financial transactions. For instance, during weekdays, there is a peak in user activity during business hours, and financial transactions are more frequent during specific time windows.\nAnomaly Detection : Temporal analysis flags an anomaly: A sudden surge in financial transactions is observed during non-business hours on a weekend. Simultaneously, there\u0026rsquo;s an unusual pattern of multiple failed login attempts to user accounts.\nCorrelation with Other Data : The system correlates temporal anomalies with other data sources, such as network logs and intrusion detection system (IDS) alerts. It discovers that the increased financial transactions coincide with a spike in outbound traffic to an unfamiliar IP address.\nContextual Analysis: # Security analysts perform contextual analysis to understand the nature of the financial transactions and the destination of the outbound traffic. They also check if this aligns with known threat indicators or attack patterns.\nConfirmation of Anomaly: # Through the analysis, it is confirmed that the temporal anomalies indicate a potential security incident. An attacker has gained unauthorized access to user accounts, attempting fraudulent financial transactions during a time when security defences might be less vigilant.\nResponse: # The security team takes immediate action to contain the incident. They block unauthorized transactions, reset compromised user accounts, and implement additional controls to prevent further unauthorized access.\nIn this example, temporal analysis plays a crucial role in detecting unusual patterns of financial transactions and login attempts, allowing the organization to respond promptly to a potential security threat.\nEnvironmental Context # Considering the broader environmental context is essential. This includes factors like the industry in which an organization operates, geopolitical events, and regulatory requirements. Understanding the external context helps in prioritizing threats and aligning cybersecurity efforts with the specific risks an organization faces.\nHere\u0026rsquo;s an example illustrating how environmental context can impact cybersecurity:\nScenario: # A multinational healthcare organization stores sensitive patient information on its servers and operates in a highly regulated industry.\nEnvironmental Context Analysis: # Industry Regulations : The healthcare industry is subject to strict data protection regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States. The organization recognizes that any security incidents leading to data breaches could result in severe legal and financial consequences.\nGeopolitical Events : There is a surge in cyber-espionage activities targeting healthcare organizations due to geopolitical tensions. Recent news reports highlight increased hacking attempts and data theft in the industry.\nEmerging Cyber Threats : The organization monitors threat intelligence feeds indicating a rise in ransomware attacks targeting healthcare institutions. Recent incidents in the industry have resulted in data encryption and ransom demands.\nInternal Changes : The organization is undergoing a digital transformation, implementing new technologies to enhance patient care. This introduces additional attack surfaces and potential vulnerabilities that need to be addressed.\nResponse Based on Environmental Context : # Recognizing the environmental context, the organization takes several strategic cybersecurity measures:\nEnhanced Regulatory Compliance : The cybersecurity strategy places a strong emphasis on compliance with healthcare regulations. The organization invests in solutions that specifically address HIPAA requirements, such as encryption and access controls.\nGeopolitical Threat Mitigation : The security team increases monitoring and implements additional controls to defend against cyber-espionage activities targeting healthcare organizations. They collaborate with industry peers to share threat intelligence and best practices.\nRansomware Preparedness : Given the rising threat of ransomware, the organization conducts regular training for employees on recognizing phishing attempts and establishes robust backup and recovery processes to minimize the impact of a potential ransomware attack.\nAdaptive Security Measures : The cybersecurity strategy acknowledges the evolving nature of the organization\u0026rsquo;s digital landscape. Security controls are regularly updated to accommodate new technologies introduced during the digital transformation, ensuring a proactive approach to emerging threats.\nIn this example, environmental context plays a pivotal role in shaping the organization\u0026rsquo;s cybersecurity strategy, allowing it to effectively navigate industry-specific challenges and emerging threats.\nImplementation of Contextual Analysis # Security Information and Event Management (SIEM) Systems # SIEM systems play a central role in contextual analysis by aggregating and analyzing security data from various sources. These platforms use correlation rules and behavioural analytics to identify potential threats. The integration of threat intelligence feeds further enhances their ability to provide context to security events.\nIt is a comprehensive solution designed to provide a holistic view of an organization\u0026rsquo;s information security. It combines Security Information Management (SIM) and Security Event Management (SEM) into one integrated platform.\nThe primary goal of an SIEM system is to provide real-time analysis of security alerts generated throughout an organization\u0026rsquo;s IT infrastructure.\nHere are the key components and functionalities of an SIEM system:\nData Collection : SIEM systems collect and aggregate log data generated throughout the organization\u0026rsquo;s technology infrastructure, from host systems and applications to network and security devices.\nNormalization and Correlation : The collected data is normalized to a standard format, allowing for more straightforward analysis. SIEM tools also correlate related events to help identify patterns and potential security threats.\nAlerting and Notification : SIEM systems can generate alerts in real-time or near real-time, notifying security administrators of potential security incidents or policy violations. Alerts are often prioritized based on severity.\nDashboards and Reporting : SIEM solutions provide dashboards and reports that offer insights into the security status of an organization. These visualizations help security professionals understand the overall security posture, trends, and potential areas of concern.\nIncident Response : SIEM systems assist in incident response by providing detailed information about security incidents, helping security teams investigate and mitigate threats more effectively.\nCompliance Monitoring : Many organizations use SIEM tools to meet regulatory compliance requirements by monitoring and reporting on activities that are subject to specific regulations.\nUser and Entity Behavior Analytics (UEBA) : Some SIEM solutions incorporate UEBA, which uses machine learning and statistical analysis to detect abnormal behaviour patterns among users and entities, helping to identify potential insider threats.\nIntegration with Other Security Tools : SIEM systems often integrate with other security solutions, such as intrusion detection/prevention systems, antivirus software, and vulnerability management tools, to provide a more comprehensive security posture.\nLog Retention and Storage : SIEM tools typically store log and event data for an extended period, allowing for historical analysis and compliance reporting.\nBy centralizing and analyzing security event data from across an organization\u0026rsquo;s technology infrastructure, SIEM systems play a crucial role in enhancing an organization\u0026rsquo;s ability to detect and respond to security incidents effectively. They are an essential component of a robust cybersecurity strategy.\nMachine Learning and Artificial Intelligence # Machine learning algorithms contribute significantly to contextual analysis. These technologies can analyze vast amounts of data, recognize patterns, and identify anomalies that may elude traditional rule-based systems. AI-driven tools excel in adapting to evolving threats by continuously learning from new data.\nAutomation and Orchestration # Automated responses to security incidents are a natural extension of contextual analysis. When an anomalous event is detected, automatic responses can be triggered to contain or mitigate the threat. Orchestration ensures that these automated actions are coordinated across different security tools and systems.\nIncident Response Planning # Contextual analysis is an integral part of incident response planning. Organizations should have predefined workflows that consider the context of an incident. This includes understanding the potential impact, the criticality of affected systems, and the broader implications for the business.\nChallenges and Considerations # False Positives and Negatives # One of the challenges in contextual analysis is the balance between minimizing false positives (incorrectly identifying benign events as threats) and avoiding false negatives (failing to detect actual threats). Achieving this balance requires fine-tuning correlation rules and leveraging advanced analytics.\nData Privacy and Compliance # As contextual analysis involves analyzing diverse data sources, ensuring compliance with data privacy regulations is crucial. Organizations must strike a balance between practical analysis and respecting the privacy rights of individuals.\nContinuous Monitoring # Contextual analysis is an ongoing process that requires continuous monitoring and adaptation. Cybersecurity threats evolve, and so should contextual analysis strategies. Regularly updating correlation rules, threat intelligence feeds and adjusting algorithms is essential to stay ahead of emerging threats.\nSkill and Resource Requirements # Implementing practical contextual analysis requires skilled cybersecurity professionals and adequate resources. Organizations need personnel who can interpret the context of security events and make informed decisions. Additionally, investing in advanced technologies and training programs is essential.\nConclusion # Contextual analysis is a dynamic and integral part of modern cybersecurity strategies. By understanding the broader context of security events, organizations can enhance their ability to detect, respond to, and mitigate cyber threats effectively. As cyber threats continue to evolve, contextual analysis will play a pivotal role in securing digital assets and maintaining the resilience of organizations in the face of cyber challenges.\n","date":"5 February 2024","externalUrl":null,"permalink":"/posts/contextual-analysis-in-cybersecurity/","section":"Posts","summary":"Contextual analysis in cybersecurity involves examining events, actions, or data within the broader context of an organization’s IT environment. It is a critical component of a proactive cybersecurity strategy, aiming to understand the significance of activities by considering various factors surrounding them. This multifaceted approach helps cybersecurity professionals identify and respond to potential threats effectively.\n","title":"Contextual Analysis in Cybersecurity","type":"posts"},{"content":"","date":"5 February 2024","externalUrl":null,"permalink":"/tags/devsecops/","section":"Tags","summary":"","title":"Devsecops","type":"tags"},{"content":"","date":"31 January 2024","externalUrl":null,"permalink":"/tags/horus-reticle/","section":"Tags","summary":"","title":"Horus Reticle","type":"tags"},{"content":"","date":"31 January 2024","externalUrl":null,"permalink":"/tags/milrad/","section":"Tags","summary":"","title":"MilRad","type":"tags"},{"content":"A milliradian (MilRad or mrad) is a unit of angular measurement commonly used in precision shooting, optics, and ballistics. It is based on dividing a circle into 6.283 radians (2π radians), with each radian further divided into 1,000 milliradians. The milliradian is often denoted as \u0026ldquo;mil.\u0026rdquo;\nWhere is MilRad used? Firearms and Shooting: Military and Tactical Operations: Optics and Binoculars: Ballistics and Trajectory Calculations: Search and Rescue Operations: Surveying and Navigation: Astronomy: Training and Education: The History of MilRad Origins in Mathematics: Military and Navigation: Standardisation and Adoption: What is a \u0026ldquo;Strichplatte\u0026rdquo; How to use MilRad? How to use MilRad in meters? What alternatives are available for MilRad? MOA (Minute of Angle): MOA Reticle: BDC (Bullet Drop Compensation) Reticle: Horus Reticle: Duplex Reticle: Christmas Tree Reticle: MOA/Mil Hybrid Reticle: Some examples using MilRad in meters Example 1: Range Estimation Example 2: Bullet Drop Compensation How to use MilRad in navigation? Land Navigation: Terrain Assessment: Distance Estimation: Obstacle Avoidance: Map Reading: Example Using MilRad for Obstacle Avoidance Scenario: Measurement: Distance Estimation: Distance Calculation: Decision-Making: Where is MilRad used? # Milliradians (MilRads or mils) are used in various fields, primarily for angular measurements and precision calculations. Here are some key areas where MilRads are commonly used:\nFirearms and Shooting: # MilRads are extensively used in firearms and shooting for range estimation, bullet drop compensation, and windage adjustments. Rifle scopes often feature Mil-Dot reticles, with dots or hash marks spaced at 1 MilRad intervals, aiding shooters in making accurate adjustments.\nMilitary and Tactical Operations: # MilRads are widely employed in military applications for sniper training, precision shooting, and artillery calculations. The standardised angular measurement provides consistency and ease of use for military personnel.\nOptics and Binoculars: # MilRads are integrated into the reticles of optics, including binoculars and spotting scopes, to assist with ranging and target observation. They are beneficial in situations where precision measurements are essential.\nBallistics and Trajectory Calculations: # MilRads are used in ballistics to calculate bullet trajectories, especially for long-range shooting. Shooters can use MilRads to adjust bullet drop and windage based on the characteristics of the ammunition and rifle.\nSearch and Rescue Operations: # MilRads can be used in search and rescue operations to estimate distances and navigate challenging terrains. The angular measurement aids in making accurate judgments about the location of objects or individuals.\nSurveying and Navigation: # MilRads can be used for angular measurements and mapping in surveying and navigation. They provide a standardised unit for precision work in fields where accurate angles are crucial.\nAstronomy: # MilRads are used in astronomy for angular measurements and observations. Astronomers may use MilRads to measure the apparent sizes of celestial objects or angular separations between them.\nTraining and Education: # MilRads are a standard part of training programs for sharpshooters, snipers, and other professionals requiring precision shooting skills. Educational materials for these fields often include MilRad-based calculations and exercises.\nMilRads finds applications where accurate angular measurements are necessary, particularly in activities that demand precision and consistency, such as shooting sports, military operations, and fields requiring precise navigational or observational data.\nThe History of MilRad # The concept of the milliradian (MilRad or mrad) has its roots in angular measurement, trigonometry, and navigation, and its history predates its widespread use in the field of firearms and optics. A milliradian is a unit of angular measurement equal to one-thousandth of a radian, and it\u0026rsquo;s widely used in various applications beyond weapons. Here\u0026rsquo;s a brief history of the MilRad:\nOrigins in Mathematics: # The radian, a unit of angular measurement, has a long history in mathematics. The radian is defined as the angle subtended when the radius of a circle is laid along the circumference, and it\u0026rsquo;s based on the concept of the radius being equal to the arc length. The milliradian is a smaller unit with 1,000 milliradians in one radian.\nMilitary and Navigation: # For centuries, angular measurement has been crucial in military applications, navigation, and artillery calculations. Using angular units for accurate targeting and measurement is deeply rooted in these fields.\nStandardisation and Adoption: # The use of the milliradian became more standardised, and its adoption in military and tactical applications increased. Military organisations and shooting communities began recognising the advantages of using a standardised angular unit for ranging and adjustments. In modern times, the milliradian is widely accepted and utilised in the firearms and optics industry. Many scopes, especially those designed for long-range shooting and precision applications, feature MilRad markings in their reticles for easy and consistent angular measurements. The use of the milliradian is not limited to any particular region or country. It has become a globally accepted standard.\nThe milliradian\u0026rsquo;s history is intertwined with the development of angular measurement in mathematics, its application in military and navigation, and its subsequent adoption and standardisation in optics for precision shooting.\nWhat is a \u0026ldquo;Strichplatte\u0026rdquo; # A \u0026ldquo;Strichplatte\u0026rdquo; is a German term that translates to \u0026ldquo;grid reticle\u0026rdquo; in English. A Strichplatte typically refers to a reticle with a grid-like pattern of markings, lines, or dots in optics and firearms. The purpose of a Strichplatte is to provide a visual reference for precise measurements and adjustments.\nWhile \u0026ldquo;Strichplatte\u0026rdquo; is more commonly associated with German-speaking regions, similar reticle designs with grid patterns are found in optics produced by various manufacturers worldwide. These reticles are designed to assist shooters in making accurate and consistent adjustments, especially in precision shooting at varying distances.\nHow to use MilRad? # Here\u0026rsquo;s a brief guide on how to use a MilRad:\nUnderstanding MilRadian :\nMilRadian is an angular measurement that is often denoted as \u0026ldquo;mil\u0026rdquo; or \u0026ldquo;mrad.\u0026rdquo; One mil represents an angle where the arc length equals the radius. In practical terms, one mil is approximately 3.6 inches at 100 yards.\nMil-Dot Reticle :\nMany scopes feature a Mil-Dot reticle, where the crosshairs have small dots or hash marks spaced at one mil intervals. The Mil-Dot system can be used for range estimation, holdovers, and windage corrections.\nRange Estimation :\nTo estimate the distance to a target using a Mil-Dot reticle, you need to know the size of the target in relation to the mils it subtends.\nThe formula for range estimation is:\n\u0026ldquo;Distance (in yards) = Size of Target (in yards) / Size of Target in Mils \u0026quot;\nIf the target\u0026rsquo;s size is known and it spans, for example, two mils, and the target is 36 inches wide, the range would be 100 yards (36 / 2 = 18 yards, and 18 * 100 = 1800).\nAdjusting for Bullet Drop :\nThe Mil-Dot system can help you adjust your scope when dealing with bullet drops. If you know the drop value in mils for a specific distance, you can use the Mil-Dot reticle to compensate for bullet drop by holding over or adjusting your scope accordingly.\nWindage Corrections :\nMilRadian can also be used to compensate for wind drift. Windage correction is often expressed in mils, and you can utilise the Mil-Dot reticle to estimate the wind\u0026rsquo;s effect on the bullet\u0026rsquo;s trajectory.\nPractice and Consistency :\nUsing MilRad effectively requires practice. Regularly shoot at known distances and use your Mil-Dot reticle to make adjustments. Consistency is key. Ensure your measurements and calculations are accurate, and always use the same unit of measure for target size and adjustments.\nKnow Your Optics :\nFamiliarise yourself with your specific scope\u0026rsquo;s Mil-Dot reticle. Different scopes may have slightly different suspensions, so it\u0026rsquo;s essential to know the details of your equipment.\nRemember, MilRadian is a versatile system, but it requires practice and an understanding of your specific optics and firearm characteristics. Always adhere to firearm safety rules, and if possible, seek guidance from experienced shooters or instructors.\nHow to use MilRad in meters? # Using MilRadian (MilRad) in meters is similar to using it in yards, but the conversion factors change because one meter is approximately 1.094 yards. Here\u0026rsquo;s a guide on how to use MilRad for range estimation and adjustments in meters:\nUnderstand MilRad :\nOne MilRad is an angular measurement where the arc length equals the radius. In practical terms, 1 MilRad is approximately 1 meter at 1000 meters. Many scopes feature a Mil-Dot reticle with dots or hash marks spaced at 1 MilRad intervals.\nRange Estimation:\nThe formula for range estimation in meters is:\n\u0026ldquo;Distance (in meters) = Size of Target (in meters) / Size of Target in MilRads \u0026quot;\nIf the target\u0026rsquo;s size is known and it spans, for example, 2 MilRads, and the target is 1.5 meters wide, the range would be 750 meters (1.5 / 2 = 0.75 meters, and 0.75 * 1000 = 750 meters).\nConversion Factors :\nRemember the conversion factor from yards to meters: 1 meter ≈ 1.094 yards. If you\u0026rsquo;re using the information provided in yards, convert it to meters before applying the MilRad calculations.\nApplying MilRad in meters involves using the same principles as in yards, with adjustments made for the difference in unit conversion.\nWhat alternatives are available for MilRad? # While the MilRad (Milliradian) is a widely used unit of angular measurement, there are alternative systems and units for similar purposes in precision shooting and optics. Some of the alternatives include:\nMOA (Minute of Angle): # MOA is another unit of angular measurement commonly used in firearms and optics. One MOA equals 1/60th of a degree, translating to approximately 1.047 inches at 100 yards. This unit is often used for making adjustments and estimating angles.\nMOA Reticle: # Similar to the Mil-Dot reticle, some scopes feature a reticle with markings spaced at 1 MOA intervals. MOA reticles are used for range estimation, holdovers, and windage adjustments, similar to Mil-Dot reticles.\nBDC (Bullet Drop Compensation) Reticle: # Some scopes have a BDC reticle to compensate for bullet drop at specific distances. Instead of relying on angular measurements like MilRad or MOA, BDC reticles often have hash marks or aiming points calibrated for specific ballistics and bullet trajectories.\nHorus Reticle: # Horus reticles are complex designs incorporating various markings and grids for precise ranging and holdover calculations. These reticles provide a more sophisticated solution for long-range shooting and can include features like windage holds and moving target leads.\nDuplex Reticle: # The duplex reticle is a simple crosshair design with thicker outer and thinner inner lines. While not specifically designed for angular measurements, it provides a precise aiming point and is often used for general shooting at various distances.\nChristmas Tree Reticle: # This reticle design includes a series of horizontal and vertical hash marks that resemble a Christmas tree. It is designed to assist with windage and elevation holds, especially in dynamic shooting scenarios.\nMOA/Mil Hybrid Reticle: # Some scopes feature a combination of MOA and MilRad markings in the same reticle to cater to shooters familiar with both systems.\nThe choice between MilRad and alternatives often comes down to personal preference, familiarity, and specific requirements. Choosing a system that aligns with your shooting style, the shooting you do, and the reticle design that you find most comfortable and practical for your needs is essential.\nSome examples using MilRad in meters # Certainly! Let\u0026rsquo;s go through a couple of examples of how MilRads might be used in meters for range estimation and adjustments:\nExample 1: Range Estimation # Suppose you have a target that measures 1.5 meters and spans 2 MilRads in your scope\u0026rsquo;s reticle.\n\u0026ldquo;Distance (in meters) = Size of Target (in meters) / Size of Target in MilRads \u0026quot;\nDistance = 1.5 meters / 2 MilRads = 0.75 meters per MilRad\nSo, the estimated distance to the target is 0.75 meters per MilRad. If you have 2 MilRads, the distance would be:\nDistance = 0.75 meters/MilRad * 2 MilRads = 1.5 meters\nExample 2: Bullet Drop Compensation # Let\u0026rsquo;s say your rifle scope\u0026rsquo;s reticle has markings for bullet drop in MilRads, and you need to compensate for a decline of 5 MilRads at a distance of 300 meters.\n“Bullet Drop (in meters) = Bullet Drop in MilRads * Distance (in meters) ”\nBullet Drop = 5 MilRads * 300 meters = 150 meters\nYou would need to hold your aim 150 meters above the target to compensate for the bullet drop at a distance of 300 meters.\nThese examples illustrate how MilRads can be used in practical scenarios for both range estimation and making adjustments for bullet drop. Remember that these calculations depend on the specific characteristics of your scope\u0026rsquo;s reticle and your ammunition\u0026rsquo;s ballistics. Refer to your scope\u0026rsquo;s user manual and consult with experts if needed.\nHow to use MilRad in navigation? # While MilRads (milliradians) are not commonly used for navigation in the traditional sense, they can be applied in specific navigation scenarios, particularly in situations where precise angular measurements are necessary. Here are a few examples of how MilRads might be used for navigation:\nLand Navigation: # In situations where traditional compass bearings are not available or practical, MilRads can be used for navigation. By measuring angles relative to a known point or landmark, you can navigate with the precision provided by angular measurements.\nTerrain Assessment: # MilRads can be utilised to estimate the steepness of the terrain. For example, if you observe a slope in the distance and can measure the angle using MilRads, you may gain insights into the difficulty of traversing that terrain.\nDistance Estimation: # Similar to its application in shooting, MilRads can be used for estimating distances to objects or landmarks. If you know the size of an object and its angular size in MilRads, you can use trigonometry to calculate the distance.\nObstacle Avoidance: # MilRads can help in assessing obstacles or barriers. By measuring the angles between your position and obstacles, you can plan alternative routes or navigate around obstacles with precision.\nMap Reading: # While not a standard method, MilRads can be used in conjunction with maps for navigation. If you have a map with detailed angular markings, you could use MilRads to align the map with your surroundings and determine your location more accurately.\nIt\u0026rsquo;s important to note that while MilRads offer precision in angular measurements, their use for navigation is less widespread than traditional methods such as degrees or compass bearings. MilRads are more commonly associated with shooting sports, military applications, and optics. If you are considering using MilRads for navigation, ensure that you are well-versed in angular measurements and trigonometry and be aware of the limitations and potential sources of error in this approach.\nExample Using MilRad for Obstacle Avoidance # Let\u0026rsquo;s consider an example of using MilRads for obstacle avoidance in kilometres:\nScenario: # Imagine you are navigating through rugged terrain, and you come across a steep incline. You want to assess the steepness of the slope using MilRads to decide whether it\u0026rsquo;s safe to climb.\nMeasurement: # Using your MilRad-based equipment, such as a rangefinder with MilRad reticle or a MilRad-equipped compass, measure the angle of the slope relative to your position. Let\u0026rsquo;s say the angle measures 3 MilRads.\nDistance Estimation: # You know that the slope starts at a certain point, and you want to estimate how far away it is before deciding. Let\u0026rsquo;s assume you can calculate the size of a recognisable feature on the slope to be 10 meters.\nDistance Calculation: # Use the MilRad formula for distance estimation:\n\u0026#34;**Distance (in kilometres) = Size of Target (in meters) / Size of Target in MilRads\u0026#34;** In this case: Distance = 10 meters / 3 MilRads ≈ 3.33 kilometres\nDecision-Making: # You\u0026rsquo;ve estimated that the steep slope is approximately 3.33 kilometres away. Based on this information and your navigation goals, you can decide whether to proceed or find an alternative route.\n","date":"31 January 2024","externalUrl":null,"permalink":"/posts/milrad-a-birdeye-view/","section":"Posts","summary":"A milliradian (MilRad or mrad) is a unit of angular measurement commonly used in precision shooting, optics, and ballistics. It is based on dividing a circle into 6.283 radians (2π radians), with each radian further divided into 1,000 milliradians. The milliradian is often denoted as “mil.”\n","title":"MilRad - a BirdEye View","type":"posts"},{"content":"","date":"31 January 2024","externalUrl":null,"permalink":"/tags/moa/","section":"Tags","summary":"","title":"MOA","type":"tags"},{"content":"","date":"31 January 2024","externalUrl":null,"permalink":"/tags/strichplatte/","section":"Tags","summary":"","title":"Strichplatte","type":"tags"},{"content":"","date":"10 January 2024","externalUrl":null,"permalink":"/tags/cwe/","section":"Tags","summary":"","title":"CWE","type":"tags"},{"content":"CWE stands for Common Weakness Enumeration. It is a community-developed list of software and hardware weakness types that can serve as a common language for describing, sharing, and identifying security vulnerabilities in software systems. CWE aims to provide a standardized way of identifying and categorizing vulnerabilities, making it easier for software developers, testers, and security professionals to discuss and address security issues.\nWhy Do We Need A Common Weakness Enumeration? Standardized Language: Improved Awareness: Efficient Communication: Systematic Analysis: Security Education and Training: Tool Integration: Vulnerability Management: Community Collaboration: Regulatory Compliance: Details about Common Weakness Enumeration Development and Maintenance: Standardized Identifiers: Categories and Views: Relationships and Mappings: Community Involvement: Software Assurance: Integration with other Standards: Education and Training: The main components of the CWE structure: An Example for the Java-World CWE-129: Improper Validation of Array Index Description: Example in Java: A List Of Example CWE´s Why Do We Need A Common Weakness Enumeration? # Common Weakness Enumeration (CWE) serves several essential purposes in cybersecurity and software development. Here are some key reasons why CWE is needed:\nStandardized Language: # CWE provides a standardized and common language for describing and categorizing software and hardware weaknesses. This common terminology helps improve communication among developers, testers, and security professionals, fostering a shared understanding of vulnerabilities.\nImproved Awareness: # CWE raises awareness about common weaknesses and vulnerabilities in software systems. This awareness is essential for developers and security practitioners to proactively identify and address potential issues during the software development life cycle.\nEfficient Communication: # Using unique identifiers (CWE IDs) for each weakness allows for efficient communication and reference. When discussing vulnerabilities, using CWE IDs ensures clarity about which specific weakness is being referred to.\nSystematic Analysis: # CWE organizes weaknesses into categories and views, enabling a systematic analysis of vulnerabilities. This organization helps identify patterns and trends in security issues, allowing for more effective risk assessment and mitigation strategies.\nSecurity Education and Training: # CWE is used as a foundation for security education and training programs. Developers can enhance their skills and knowledge by understanding common weaknesses, bettering them to write secure code and design robust systems.\nTool Integration: # Many security tools and systems use CWE to categorize and report vulnerabilities. This integration streamlines identifying and remedying weaknesses by providing a standardized way for different tools to communicate and share information about security issues.\nVulnerability Management: # CWE contributes to effective vulnerability management by providing a comprehensive list of weaknesses. This allows organizations to prioritize and address vulnerabilities based on their severity and potential impact on the system.\nCommunity Collaboration: # The collaborative development and maintenance of CWE involve input from a wide range of stakeholders, including researchers, industry professionals, and organizations. This collaborative approach ensures that CWE remains comprehensive, relevant, and reflective of the evolving landscape of software vulnerabilities.\nRegulatory Compliance: # Some regulatory frameworks and standards reference or require CWE to identify and manage software weaknesses. Compliance with these standards may be necessary for organizations operating in specific industries or regions.\nIn summary, Common Weakness Enumeration is crucial in enhancing cybersecurity practices by providing a standardized and widely accepted framework for identifying, categorizing, and addressing software weaknesses and vulnerabilities. It contributes to a more secure software development and deployment ecosystem by promoting awareness, communication, and collaboration within the cybersecurity community.\nDetails about Common Weakness Enumeration # Development and Maintenance: # CWE is developed and maintained by the MITRE Corporation, a not-for-profit organization that operates Federally Funded Research and Development Centers (FFRDCs) in the United States.\nStandardized Identifiers: # Each weakness in the CWE list is assigned a unique CWE ID identifier. This helps in unambiguously referring to specific weaknesses when discussing vulnerabilities.\nCategories and Views: # CWE organizes weaknesses into categories and views. Categories group similar weaknesses together, making understanding and addressing issues systematically easier. Views provide different perspectives on the data, allowing users to focus on specific aspects of weaknesses.\nRelationships and Mappings: # CWE captures relationships between weaknesses, showing how they might be related or lead to other vulnerabilities. It also provides mappings to other relevant standards and frameworks, facilitating integration with different tools and processes.\nCommunity Involvement: # The development of CWE involves collaboration with the broader cybersecurity community, including researchers, developers, and security professionals. Input is collected to ensure that the list remains comprehensive and up-to-date.\nSoftware Assurance: # CWE is part of a broader initiative focused on improving software assurance. Providing a common language for describing weaknesses contributes to building more secure and robust software systems.\nIntegration with other Standards: # CWE is often used alongside standards like the Common Vulnerability Scoring System (CVSS) and the Common Vulnerability and Exposure (CVE) system. Together, These standards help assess, score, and address vulnerabilities in a coordinated manner.\nEducation and Training: # CWE is used for educational purposes, helping individuals and organizations better understand common software weaknesses and vulnerabilities. Training materials and resources are developed to support this educational aspect.\nIn summary, Common Weakness Enumeration plays a crucial role in standardizing the identification and classification of software weaknesses, providing a foundation for improving cybersecurity practices across the software development and security communities.\nThe main components of the CWE structure: # Entry Point :\nYou have the entry points at the top level of the hierarchy. These represent high-level categories or groups of weaknesses. Examples of entry points include \u0026ldquo;CWE-119: Improper Restriction of Operations within the Bounds of a Memory Buffer\u0026rdquo; and \u0026ldquo;CWE-200: Exposure of Sensitive Information to an Unauthorized Actor.\u0026rdquo;\nCategories :\nEntry points are further divided into categories. Categories group related weaknesses together based on similar characteristics or types of vulnerabilities. For example, under \u0026ldquo;CWE-119,\u0026rdquo; you might have categories like \u0026ldquo;CWE-120: Buffer Copy without Checking Size of Input (\u0026lsquo;Classic Buffer Overflow\u0026rsquo;)\u0026rdquo; or \u0026ldquo;CWE-121: Stack-based Buffer Overflow.\u0026rdquo;\nWeaknesses :\nCategories are broken down into specific weaknesses. Each weakness has a unique (CWE ID) identifier and detailed description. For example, under \u0026ldquo;CWE-120,\u0026rdquo; you might have particular weaknesses like \u0026ldquo;CWE-120: Classic Buffer Overflow in strcat()\u0026rdquo; or \u0026ldquo;CWE-121: Stack-based Buffer Overflow in strcpy()\u0026rdquo;.\nRelationships :\nCWE also captures relationships between weaknesses. This includes hierarchical relationships, where a more general weakness encompasses more specific weaknesses. Additionally, it identifies other types of relationships, such as dependencies, mappings to other standards, and variations.\nViews :\nCWE provides different views to offer alternative perspectives on the data. Views may focus on specific aspects of weaknesses, and users can switch between views to examine the information from different angles.\nMitigations :\nEach weakness entry includes information on potential mitigations or ways to address and prevent the weakness. This helps users understand how to secure their software against specific vulnerabilities.\nAn Example for the Java-World # One common weakness is the improper validation of array indices, which can lead to various security issues, including buffer overflows. CWE-129 specifically addresses this issue.\nCWE-129: Improper Validation of Array Index # Description: # Improper validation of array indices in Java can lead to out-of-bounds read or write access, potentially causing unexpected behaviour, crashes, or security vulnerabilities.\nExample in Java: # public class ArrayIndexExample { public static void main(String[] args) { int[] numbers = {1, 2, 3, 4, 5}; // Incorrect: Accessing an array element without proper bounds checking int index = 10; int value = numbers[index]; // This can lead to ArrayIndexOutOfBoundsException // Correct: Adding proper bounds checking to prevent array index issues if (index \u0026gt;= 0 \u0026amp;\u0026amp; index \u0026lt; numbers.length) { value = numbers[index]; System.out.println(\u0026#34;Value at index \u0026#34; + index + \u0026#34;: \u0026#34; + value); } else { System.out.println(\u0026#34;Invalid index: \u0026#34; + index); } } } In this example, the improper validation of the array index occurs when trying to access an element at the index **10** without checking whether it is within the array\u0026rsquo;s bounds. This can result in an **ArrayIndexOutOfBoundsException** at runtime. The corrected version includes proper bounds checking to ensure the index is within the valid range before accessing the array.\nAddressing CWEs like this one is crucial for writing secure Java code and avoiding common vulnerabilities related to array index manipulation. Keep in mind that this is just one example, and there are many other CWEs that developers should be aware of and mitigate in their Java applications.\nA List Of Example CWE´s # Certainly! Below is a list of Common Weakness Enumeration (CWE) entries that are commonly associated with security issues in Java applications. Each entry includes a brief description:\nCWE-20: Improper Input Validation\nFailure to properly validate user-supplied input can lead to various security vulnerabilities, including injection attacks (e.g., SQL injection, command injection).\nCWE-79: Improper Neutralization of Input During Web Page Generation (\u0026lsquo;Cross-site Scripting\u0026rsquo;)\nFailure to properly sanitize user input before rendering it on a web page can lead to Cross-Site Scripting (XSS) vulnerabilities.\nCWE-89: Improper Neutralization of Special Elements used in an SQL Command (\u0026lsquo;SQL Injection\u0026rsquo;)\nFailure to properly validate and sanitize input used in SQL queries can result in SQL injection vulnerabilities.\nCWE-77: Improper Neutralization of Special Elements used in a Command (\u0026lsquo;Command Injection\u0026rsquo;)\nLike SQL injection, improper user input validation in command-based operations can lead to command injection vulnerabilities.\nCWE-352: Cross-Site Request Forgery (CSRF)\nInadequate validation of requests can expose applications to Cross-Site Request Forgery attacks, where unauthorized actions are performed on behalf of a user.\nCWE-306: Missing Authentication for Critical Function\nFailing to enforce proper authentication for critical functions can lead to unauthorized access and potential security breaches.\nCWE-285: Improper Authorization\nInadequate enforcement of access controls and permissions can result in unauthorized access to sensitive data or functionality.\nCWE-311: Missing Encryption of Sensitive Data\nFailure to encrypt sensitive data during storage or transmission can lead to data exposure and privacy issues.\nCWE-732: Incorrect Permission Assignment for Critical Resource\nAssigning incorrect permissions to critical resources can result in unauthorized access and potential security vulnerabilities.\nCWE-400: Uncontrolled Resource Consumption (\u0026lsquo;Resource Exhaustion\u0026rsquo;)\nLack of proper resource management can lead to resource exhaustion attacks, where an attacker consumes system resources to disrupt service availability.\nCWE-601: URL Redirection to Untrusted Site (\u0026lsquo;Open Redirect\u0026rsquo;)\nImproper validation of user-supplied input in URL redirection can lead to open redirect vulnerabilities, potentially facilitating phishing attacks.\nCWE-502: Deserialization of Untrusted Data\nInsecure deserialization of untrusted data can lead to remote code execution and other security vulnerabilities.\nIt\u0026rsquo;s important to note that the information provided here is a brief overview, and developers should refer to the official CWE website or other authoritative sources for more in-depth information and guidance on mitigating these vulnerabilities in Java applications.\n","date":"10 January 2024","externalUrl":null,"permalink":"/posts/what-is-a-common-weakness-enumeration-cwe/","section":"Posts","summary":"CWE stands for Common Weakness Enumeration. It is a community-developed list of software and hardware weakness types that can serve as a common language for describing, sharing, and identifying security vulnerabilities in software systems. CWE aims to provide a standardized way of identifying and categorizing vulnerabilities, making it easier for software developers, testers, and security professionals to discuss and address security issues.\n","title":"What is a Common Weakness Enumeration - CWE","type":"posts"},{"content":"","date":"22 December 2023","externalUrl":null,"permalink":"/tags/bushcrafting/","section":"Tags","summary":"","title":"Bushcrafting","type":"tags"},{"content":"","date":"22 December 2023","externalUrl":null,"permalink":"/tags/magnetic-anomalies/","section":"Tags","summary":"","title":"Magnetic Anomalies","type":"tags"},{"content":"Magnetic anomalies refer to variations in the Earth\u0026rsquo;s magnetic field strength at different locations on the Earth\u0026rsquo;s surface. These anomalies interest scientists and researchers in various fields, including geophysics, geology, and environmental science. This comprehensive exploration will explore magnetic anomalies\u0026rsquo; nature, causes, measurement methods, and significance.\nIntroduction: # The Earth\u0026rsquo;s magnetic field is a complex and dynamic force crucial in various geophysical processes. It is generated by the movement of molten iron and nickel in the Earth\u0026rsquo;s outer core through a process known as the geodynamo. The resulting magnetic field extends around the planet and interacts with geological structures, materials, and procedures.\nIntroduction: Types of Magnetic Anomalies: Regional Magnetic Anomalies: Characteristics of Regional Magnetic Anomalies: Causes of Regional Magnetic Anomalies: Measurement and Mapping: Significance: Conclusion - Regional Magnetic Anomalies: Local Magnetic Anomalies: Characteristics of Local Magnetic Anomalies: Causes of Local Magnetic Anomalies: Measurement Techniques: Significance of Local Magnetic Anomalies: Case Studies: Challenges and Future Directions: Conclusion - Local Magnetic Anomalies: What is the difference between Local Magnetic Anomalies and Regional Magnetic Anomalies? Spatial Scale: Spatial Extent: Causes and Geological Significance: Measurement and Detection: Examples: Application Areas: Challenges and Future Directions: Conclusion: Wenn Du den Artikel auf deutsch lesen möchtest\u0026hellip;\nhttps://outdoorskills.blog/?p=334\nTypes of Magnetic Anomalies: # Magnetic anomalies can be broadly categorised into positive and negative anomalies. Positive anomalies indicate an above-average magnetic field strength, while negative anomalies suggest a below-average magnetic field strength. These anomalies can be further classified into regional and local anomalies based on their spatial extent.\nRegional Magnetic Anomalies: # Regional magnetic anomalies are variations in the Earth\u0026rsquo;s magnetic field strength over relatively large geographical areas. These anomalies extend beyond local variations and cover regional scales, often reflecting broader geological features and tectonic processes. Understanding regional magnetic anomalies is essential for understanding the Earth\u0026rsquo;s crustal structure, composition, and tectonic history. Here, we will explore regional magnetic anomalies\u0026rsquo; characteristics, causes, and significance.\nCharacteristics of Regional Magnetic Anomalies: # Large Spatial Scale : Regional magnetic anomalies cover extensive geographic areas, often spanning hundreds of kilometres. These anomalies are not confined to specific geological structures but are associated with the overall geological framework of a region.\nGradual Changes : Unlike local magnetic anomalies, which can exhibit rapid and abrupt changes, regional anomalies typically show more gradual variations in magnetic field strength. These changes are often linked to the regional geological setting.\nTectonic Influence : Tectonic processes, such as the movement of tectonic plates and the associated creation and destruction of crustal material, play a significant role in generating regional magnetic anomalies.\nCrustal Thickness and Composition : Variations in the thickness and composition of the Earth\u0026rsquo;s crust contribute to regional magnetic anomalies. Thicker crusts produce positive anomalies, while thinner crusts result in negative anomalies.\nTectonic Plate Boundaries : Regional magnetic anomalies are commonly found near tectonic plate boundaries. These anomalies are linked to the geological processes occurring at these boundaries, such as subduction, spreading, and crustal deformation.\nCauses of Regional Magnetic Anomalies: # Several geological factors contribute to the formation of regional magnetic anomalies:\nCrustal Differentiation : Variations in the composition of the Earth\u0026rsquo;s crust, including the presence of magnetic minerals such as magnetite, contribute to regional magnetic anomalies. Differentiation processes during the Earth\u0026rsquo;s history can result in the uneven distribution of magnetic materials.\nTectonic Activity : Tectonic processes, such as the collision and subduction of tectonic plates, influence the distribution of magnetic anomalies. For example, mountain ranges and new crust formation at mid-ocean ridges can lead to regional magnetic variations.\nSedimentary Basins : The presence of sedimentary basins can contribute to regional magnetic anomalies. Sedimentary rocks are generally less magnetic than crystalline rocks, and variations in the thickness and composition of sedimentary cover can influence regional magnetic patterns.\nMantle Contributions : Interactions with the Earth\u0026rsquo;s upper mantle influence some regional magnetic anomalies. Mantle processes, such as the movement of mantle plumes, can affect the magnetic properties of overlying crustal rocks.\nMeasurement and Mapping: # Measuring and mapping regional magnetic anomalies involve the use of various techniques:\nSatellite and Airborne Magnetometry : Remote sensing technologies, including satellite and airborne magnetometers, are employed to survey large regions efficiently. These instruments measure the Earth\u0026rsquo;s magnetic field from above, providing a broad overview of regional magnetic patterns.\nAeromagnetic Surveys : Aeromagnetic surveys involve flying specialised instruments over large areas to collect magnetic data. These surveys are beneficial for identifying regional magnetic trends and patterns.\nCompilation of Magnetic Maps : Data from ground-based magnetometers and remote sensing platforms are compiled to create magnetic maps. These maps depict the distribution of magnetic anomalies across regional scales.\nSignificance: # Understanding regional magnetic anomalies has several important implications:\nTectonic Studies : Regional magnetic anomalies contribute valuable information to tectonic studies, helping researchers decipher a region\u0026rsquo;s geological history and evolution. Anomalies near plate boundaries, for instance, provide insights into ongoing tectonic processes.\nResource Exploration : Identifying regional magnetic anomalies is crucial for resource exploration. These anomalies can highlight areas with potential mineral deposits, aiding in discovering valuable resources such as ore bodies and hydrocarbons.\nGeological Mapping : Regional magnetic data contribute to geological mapping efforts, allowing researchers to delineate major geological structures, boundaries, and trends. This information is fundamental for understanding the geological framework of a region.\nEnvironmental Studies : Changes in regional magnetic anomalies can indicate ecological factors such as soil composition, erosion patterns, and land-use changes. Monitoring these changes supports environmental studies and assessments.\nConclusion - Regional Magnetic Anomalies: # Regional magnetic anomalies provide a wealth of information about the Earth\u0026rsquo;s crustal composition, tectonic history, and geological processes at a large scale. By analysing these anomalies, scientists and researchers can gain valuable insights into the dynamic nature of the Earth\u0026rsquo;s interior and its ongoing geological evolution. The interdisciplinary approach, combining geophysics, geology, and remote sensing technologies, continues to enhance our understanding of regional magnetic anomalies and their significance in various scientific endeavours.\nLocal Magnetic Anomalies: # Local magnetic anomalies are variations in the Earth\u0026rsquo;s magnetic field that occur over more minor spatial scales, typically confined to specific geographic locations. Unlike regional magnetic anomalies, which cover larger areas and are associated with broader geological features, local anomalies are more localised and often tied to specific subsurface geological structures, mineral deposits, or human activities. Here, we will explore local magnetic anomalies\u0026rsquo; characteristics, causes, measurement methods, and significance.\nCharacteristics of Local Magnetic Anomalies: # Small Spatial Scale : Local magnetic anomalies are limited to relatively small geographic areas, ranging from a few metres to several kilometres. They are often associated with specific geological features or anthropogenic activities.\nRapid Changes : Unlike regional anomalies, local anomalies can exhibit more rapid and abrupt changes in magnetic field strength. This is because they are often linked to localised geological structures or the presence of magnetic materials.\nGeological Features : Local anomalies are commonly associated with specific geological features such as faults, igneous intrusions, mineral deposits, and other subsurface structures that influence the magnetic properties of the surrounding rocks.\nHuman-Induced Anomalies : Certain human activities, such as using metal structures, landfills, and excavation, can create local magnetic anomalies. These anomalies are often short-lived and can be related to the presence of ferrous materials.\nCauses of Local Magnetic Anomalies: # Geological Structures : Faults, fractures, and other geological structures can create local variations in magnetic field strength. These anomalies are often a result of differences in the magnetic properties of adjacent rock formations.\nMineral Deposits : Ore bodies containing magnetic minerals, such as magnetite or hematite, can produce local solid magnetic anomalies. These anomalies are valuable indicators for mineral exploration.\nIgneous Intrusions : The emplacement of igneous rocks, especially those rich in magnetic minerals, can lead to local magnetic anomalies. These intrusions alter the surrounding magnetic field due to the presence of rocks with contrasting magnetic properties.\nHuman Activities : Anthropogenic factors, including metal structures, buried utilities, and landfills, can generate local magnetic anomalies. Construction and excavation activities may also disrupt the natural magnetic field in a specific area.\nMeasurement Techniques: # Measuring and mapping local magnetic anomalies involve a range of techniques:\nGround-Based Magnetometers : Portable magnetometers are used on the Earth\u0026rsquo;s surface to measure magnetic field variations at specific locations. These instruments provide high-resolution data and are helpful for detailed investigations in small areas.\nSurveys and Transects : Researchers conduct magnetic surveys and transects over specific regions of interest to collect detailed magnetic data. This approach helps identify and characterise local anomalies associated with geological features or mineral deposits.\nMagnetic Gradiometry : Magnetic gradiometers measure spatial variations in magnetic field gradients, providing additional information about the distribution and intensity of local anomalies. This method enhances the detection of subtle magnetic changes.\nSignificance of Local Magnetic Anomalies: # Mineral Exploration : Local magnetic anomalies are crucial indicators for mineral exploration. They help geologists identify potential ore bodies and guide resource exploration efforts.\nGeotechnical Studies : Understanding local magnetic anomalies is essential for geotechnical studies, especially in construction and infrastructure development. Identifying subsurface structures can mitigate potential challenges during engineering projects.\nArchaeological Investigations : Archaeologists use magnetic anomaly mapping to locate buried artefacts and archaeological features. Magnetic surveys assist in non-invasive investigations of historical sites.\nEnvironmental Assessments : Monitoring local magnetic anomalies can be valuable for environmental assessments, especially in areas where human activities may influence the magnetic field. This includes detecting buried waste or assessing the impact of construction projects.\nCase Studies: # Iron Ore Deposits in Kiruna, Sweden : The Kiruna mine in Sweden, one of the largest iron ore mines globally, was discovered through magnetic anomaly mapping. The local solid magnetic anomaly led to the identification of extensive iron ore deposits.\nArchaeological Site Mapping : Magnetic anomaly studies have been used to map ancient structures and burial sites. For example, mapping magnetic anomalies helped identify buried structures at archaeological sites like Stonehenge.\nChallenges and Future Directions: # Challenges associated with local magnetic anomalies include:\nThe need for precise data interpretation. Distinguishing between natural and anthropogenic sources. Accounting for the influence of near-surface materials. Future research may focus on integrating magnetic data with other geophysical methods for a more comprehensive understanding of subsurface structures.\nConclusion - Local Magnetic Anomalies: # Local magnetic anomalies provide:\nValuable insights into the Earth\u0026rsquo;s subsurface. Offering information about specific geological features. Mineral deposits. Human-induced changes. The ability to identify and interpret these anomalies has practical applications in resource exploration, environmental studies, and archaeological investigations. As technology advances, the study of local magnetic anomalies will remain vital to interdisciplinary research in geophysics, geology, archaeology, and environmental science.\nWhat is the difference between Local Magnetic Anomalies and Regional Magnetic Anomalies? # Local and regional magnetic anomalies are two distinct types of variations in the Earth\u0026rsquo;s magnetic field that differ in scale, spatial extent, and geological significance. Here are the key differences between them:\nSpatial Scale: # Local Magnetic Anomalies : Limited to relatively small geographic areas, typically ranging from a few metres to several kilometres. They are associated with specific geological features, mineral deposits, or human activities.\nRegional Magnetic Anomalies :\nCover larger spatial scales, often spanning hundreds of kilometres or more. Reflect broader geological features and tectonic processes on a regional or continental scale.\nSpatial Extent: # Local Magnetic Anomalies :\nConfined to specific, localised regions and often linked to individual geological structures or features.\nIt can exhibit rapid and abrupt changes in magnetic field strength.\nRegional Magnetic Anomalies :\nExtend over extensive geographic areas and are associated with the overall geological framework of a region. Show more gradual variations in magnetic field strength over larger distances.\nCauses and Geological Significance: # Local Magnetic Anomalies :\nThey are primarily caused by specific geological structures, mineral deposits, igneous intrusions, or human activities. Signify localised variations in the magnetic properties of rocks and materials. It is often crucial for mineral exploration, archaeological studies, and geotechnical assessments in small areas.\nRegional Magnetic Anomalies :\nIt is caused by broader geological processes, including variations in crustal thickness, composition, and tectonic activity. Indicate large-scale features such as tectonic plate boundaries, sedimentary basins, and mantle contributions. It is crucial for understanding regional tectonic history, geological evolution, and resource exploration over broader areas.\nMeasurement and Detection: # Local Magnetic Anomalies :\nMeasured using ground-based magnetometers, magnetic surveys, and transects over specific regions of interest. High-resolution data collection is often necessary for detailed investigations.\nRegional Magnetic Anomalies :\nThey are measured using satellite and airborne magnetometers and aeromagnetic surveys covering large areas. Remote sensing technologies are employed to capture regional magnetic trends and patterns.\nExamples: # Local Magnetic Anomalies :\nOre bodies are in a specific mine, faults are in a small region, and artefacts are buried at an archaeological site. Examples include the magnetic anomalies associated with specific geological structures or localised human activities.\nRegional Magnetic Anomalies :\nMagnetic striping along mid-ocean ridges, variations in the Earth\u0026rsquo;s magnetic field over a continental plate. Examples include anomalies that provide insights into a large region\u0026rsquo;s overall tectonic history and geological characteristics.\nApplication Areas: # Local Magnetic Anomalies :\nEssential for mineral exploration, archaeological investigations, and geotechnical studies in localised areas. They are used to detect and understand specific geological features and human-induced changes.\nRegional Magnetic Anomalies :\nCrucial for regional geological mapping, understanding tectonic processes, and identifying potential resources over broader scales. They are used in plate tectonics studies, environmental monitoring, and large-scale geological assessments.\nIn summary, the key distinction lies in the scale and spatial extent of the anomalies, with local anomalies confined to smaller regions and associated with specific features. In contrast, regional anomalies cover larger areas and reflect broader geological processes and tectonic features. Both types of anomalies contribute valuable information to understanding the Earth\u0026rsquo;s subsurface and have diverse applications across scientific disciplines.\nChallenges and Future Directions: # Despite the advancements in magnetic anomaly studies, challenges persist, such as accurately distinguishing between various geological sources of anomalies and improving the resolution of measurements. Future research may focus on integrating magnetic data with other geophysical techniques for a more comprehensive understanding of subsurface structures.\nConclusion: # Magnetic anomalies provide:\nA unique window into the Earth\u0026rsquo;s subsurface. Offering valuable insights into geological processes. Resource exploration. Environmental monitoring. The interdisciplinary nature of magnetic anomaly studies, combining aspects of geophysics, geology, and environmental science, underscores their significance in advancing our understanding of the dynamic planet we inhabit. As technology evolves, magnetic anomaly research will likely play an increasingly vital role in addressing complex scientific questions and practical applications.\n","date":"22 December 2023","externalUrl":null,"permalink":"/posts/what-are-magnetic-anomalies/","section":"Posts","summary":"Magnetic anomalies refer to variations in the Earth’s magnetic field strength at different locations on the Earth’s surface. These anomalies interest scientists and researchers in various fields, including geophysics, geology, and environmental science. This comprehensive exploration will explore magnetic anomalies’ nature, causes, measurement methods, and significance.\n","title":"What Are Magnetic Anomalies?","type":"posts"},{"content":"","date":"21 December 2023","externalUrl":null,"permalink":"/tags/inclination/","section":"Tags","summary":"","title":"Inclination","type":"tags"},{"content":" Introduction # Navigation is an ancient and essential practice that involves determining one\u0026rsquo;s position and direction relative to the Earth\u0026rsquo;s surface. In navigation, various factors and concepts play crucial roles in helping sailors, pilots, and even modern travellers reach their destinations safely and efficiently. Inclination is an integral concept to understanding the Earth\u0026rsquo;s magnetic field and its impact on navigation. In this comprehensive guide, we will delve into the concept of inclination in navigation, exploring its definition, significance, historical context, and practical applications.\nWer den Artikel in deutsch lesen möchte\u0026hellip;\nhttps://outdoorskills.blog/2023/12/21/neigung-in-der-navigation-eine-komprimierte-ubersicht/\nIntroduction What is Inclination? The Earth\u0026rsquo;s Magnetic Field Critical Components of the Earth\u0026rsquo;s Magnetic Field: Significance of Inclination in Navigation Historical Development of Inclination Studies Practical Applications of Inclination in Navigation Methods for Measuring Inclination Conclusion What is Inclination? # In the context of navigation and geomagnetism, inclination refers to the angle between the Earth\u0026rsquo;s magnetic field lines and the horizontal plane at a particular location on the Earth\u0026rsquo;s surface. In simpler terms, it represents the tilt or angle at which the magnetic field lines penetrate the Earth\u0026rsquo;s surface. This angle is critical because it directly influences how compass needles behave and how navigational instruments, such as magnetic compasses, function.\nThe Earth\u0026rsquo;s Magnetic Field # To understand inclination fully, we must first grasp the fundamental characteristics of the Earth\u0026rsquo;s magnetic field. The Earth behaves like a giant magnet with a magnetic pole in the north and a magnetic pole in the south, which are not coincident with the geographic North and South Poles. This magnetic field extends from the core of the Earth and surrounds it, shaping the behaviour of compass needles and other magnetic materials on the surface.\nCritical Components of the Earth\u0026rsquo;s Magnetic Field: # Magnetic North Pole : This is the location on the Earth\u0026rsquo;s surface where the magnetic field lines point vertically downward. It is not the same as the geographic North Pole, and its position has been known to shift over time.\nMagnetic South Pole : This is the location on the Earth\u0026rsquo;s surface where the magnetic field lines point vertically upward. It is not the same as the geographic South Pole.\nMagnetic Equator : The magnetic equator is an imaginary line that encircles the Earth equidistant from the magnetic poles. The inclination is zero along this line, meaning that the magnetic field lines are horizontal to the Earth\u0026rsquo;s surface.\nMagnetic Field Lines : These lines represent the direction and strength of the Earth\u0026rsquo;s magnetic field at different locations. The angle at which these lines intersect the Earth\u0026rsquo;s surface determines the inclination.\nSignificance of Inclination in Navigation # Inclination is a crucial parameter in navigation for several reasons:\nCompass Behavior : Inclination directly affects the behaviour of magnetic compass needles. Compasses align with the local magnetic field lines, making them essential tools for determining direction at sea, in the air, or on land.\nNavigational Accuracy : Understanding inclination allows navigators to correct compass readings to account for magnetic field strength and direction variations, ensuring accurate course plotting and navigation.\nMagnetic Anomalies : Certain geographic regions have significant variations in inclination, which can result in magnetic anomalies. These anomalies are essential in geophysical exploration, mineral prospecting, and understanding the Earth\u0026rsquo;s geological features.\nAbout Magnetic Anomalies, a separate article about this topic is here: XXX\nMagnetic Pole Location : Monitoring changes in inclination over time helps scientists track the movement of the magnetic poles, providing insights into the Earth\u0026rsquo;s core dynamics and geological processes.\nHistorical Development of Inclination Studies # The concept of inclination in navigation has a rich history, dating back to ancient civilizations. Here is a brief overview of its historical development:\nEarly Compasses : Chinese inventors are credited with developing the earliest compasses around the 4th century BCE. These compasses consisted of a magnetised lodestone floating in water, and they pointed south. While the Chinese initially used these compasses for divination and fortune-telling, they soon recognized their navigational potential.\nMarco Polo\u0026rsquo;s Observations : In the 13th century, the famous Venetian explorer Marco Polo described the use of compasses in his travelogue. He observed that compass needles did not always point north and noted that their inclination varied with geographic location.\nThe Declination-Inclination Connection : By the 16th century, European navigators, including Gerardus Mercator and Georg Hartmann, began to study the relationship between the magnetic declination (the angle between magnetic north and true north) and inclination. This work laid the foundation for understanding magnetic variation and its impact on navigation.\nModern Geomagnetism : The 19th and 20th centuries saw significant advancements in geomagnetism. Scientists like Carl Friedrich Gauss made essential contributions to measuring and understanding the Earth\u0026rsquo;s magnetic field, including inclination.\nPractical Applications of Inclination in Navigation # Inclination plays a crucial role in various aspects of navigation. Let us explore some of its practical applications:\nMarine Navigation : Magnetic compasses are still widely used for navigation on ships and boats. Mariners must account for inclination when using these compasses to set courses and calculate proper headings from magnetic readings.\nAviation : Aircraft also rely on magnetic compasses, especially when electronic navigation systems are unavailable or as a backup. Pilots must adjust for inclination to maintain accurate headings.\nGeophysical Exploration : In geological and geophysical surveys, inclination measurements identify anomalies in the Earth\u0026rsquo;s magnetic field. This information helps locate subsurface resources like minerals and oil deposits.\nArchaeology : In archaeological excavations, inclination measurements can help date ancient pottery and artefacts by examining the alignment of magnetic materials within them.\nScientific Research : Scientists use inclination data to study changes in the Earth\u0026rsquo;s magnetic field and its implications for understanding the planet\u0026rsquo;s core dynamics and geological processes.\nMethods for Measuring Inclination # Several methods are employed to measure inclination in different contexts. The choice of method depends on the specific requirements of the application. Here are some standard techniques:\nInclinometers : Inclinometers are instruments designed to measure the angle of inclination. They come in various forms, such as bubbles, pendulums, and electronic digital inclinometers.\nMagnetometers : Magnetometers measure the intensity and direction of the Earth\u0026rsquo;s magnetic field. Fluxgate magnetometers and proton precession magnetometers are commonly used to measure magnetic field parameters, including inclination.\nGyrocompasses : Gyrocompasses are advanced navigation instruments that use a gyroscope to determine true north. They are not affected by magnetic field variations, making them valuable for accurate navigation in regions with extreme inclination.\nGeophysical Surveys : Geophysicists use specialised instruments and techniques, including the magnetic gradiometer and proton magnetometer surveys, to map magnetic anomalies and determine inclination in geological exploration.\nConclusion # Inclination is a fundamental concept in navigation, intimately tied to the Earth\u0026rsquo;s magnetic field and its impact on compasses and navigational instruments. Its historical development has deep roots in the history of exploration and science. In practical terms, inclination measurements are crucial for maintaining accurate heading information in various fields, from marine and aviation navigation to geophysics and scientific research. Navigators can ensure safe and precise navigation by understanding and accounting for inclination, and scientists can gain valuable insights into the Earth\u0026rsquo;s dynamic processes. As technology and scientific knowledge advance, the study of inclination remains essential to our understanding of the Earth\u0026rsquo;s magnetic field and its role in shaping the navigation world.\n","date":"21 December 2023","externalUrl":null,"permalink":"/posts/inclination-in-navigation-a-short-overview/","section":"Posts","summary":"Introduction # Navigation is an ancient and essential practice that involves determining one’s position and direction relative to the Earth’s surface. In navigation, various factors and concepts play crucial roles in helping sailors, pilots, and even modern travellers reach their destinations safely and efficiently. Inclination is an integral concept to understanding the Earth’s magnetic field and its impact on navigation. In this comprehensive guide, we will delve into the concept of inclination in navigation, exploring its definition, significance, historical context, and practical applications.\n","title":"Inclination in Navigation: A Short Overview","type":"posts"},{"content":"","date":"21 December 2023","externalUrl":null,"permalink":"/tags/magnetic-anomalie/","section":"Tags","summary":"","title":"Magnetic Anomalie","type":"tags"},{"content":"","date":"20 December 2023","externalUrl":null,"permalink":"/tags/compass/","section":"Tags","summary":"","title":"Compass","type":"tags"},{"content":"","date":"20 December 2023","externalUrl":null,"permalink":"/tags/declination/","section":"Tags","summary":"","title":"Declination","type":"tags"},{"content":"","date":"20 December 2023","externalUrl":null,"permalink":"/tags/deviation/","section":"Tags","summary":"","title":"Deviation","type":"tags"},{"content":"Understanding the disturbance of a magnetic compass is crucial for accurate navigation. External factors like magnetic declination, ferrous objects, electromagnetic interference, temperature variations, and external forces can affect compass readings. Magnetic declination represents the angular difference between true north and magnetic north. Ferrous objects and nearby magnets can alter the local magnetic field, causing deviations. Electromagnetic interference from electronic devices may disrupt compass accuracy. Temperature variations influence materials and magnetic properties within the compass. External forces, such as vibrations and movement, can lead to temporary disturbances. To maintain accuracy, users must know these factors, hold the compass level, calibrate periodically, and avoid interference sources. Understanding and mitigating compass disturbances are essential for reliable and precise direction finding in navigation and safety-critical situations. Let\u0026rsquo;s have a look at these topics.\nWhy is it important to understand the disturbance factors? Navigation Accuracy: Safety: Emergency Situations: Proper Equipment Usage: Calibration and Correction: Educational Purposes: Technology Development: What are the disturbance variables? Magnetic Declination: How do we extract the magnetic declination from the field? How do we Use the Sun and Shadow Method to define local magnetic declination? Magnetic Deviation: Electromagnetic Interference (EMI): Ferrous (Iron) Objects: Proximity to Magnets: Tilting and Inclination: Temperature Variations: External Forces: How do I identify in the field that my magnetic compass is disturbed? To confirm if your magnetic compass is disturbed: What are the possible corrective actions? Conclusion: Wenn Du diesen Artikel auf deutsch lesen möchtest\u0026hellip;\nhttps://outdoorskills.blog/2023/12/20/navigation-kompass-und-seine-storgrosen/\nWhy is it important to understand the disturbance factors? # Understanding how a magnetic compass can be disturbed is crucial, especially in applications where precise navigation and accurate direction finding are essential. Here are some key reasons why it\u0026rsquo;s important to understand the potential disturbances of a magnetic compass:\nNavigation Accuracy: # Accurate direction finding is paramount in navigation, whether on land, at sea, or in the air. Disturbances to a magnetic compass can lead to errors in navigation, potentially causing individuals or vehicles to deviate from their intended course. Understanding and mitigating these disturbances contribute to safer and more accurate navigation.\nSafety: # Incorrect navigation can pose significant safety risks when a magnetic compass is a primary tool for orientation, such as in wilderness navigation, marine navigation, or aviation; relying on inaccurate compass readings can lead to getting lost, entering hazardous areas, or encountering obstacles.\nEmergency Situations: # The ability to navigate accurately is crucial during emergencies or adverse weather conditions. Understanding how external factors, such as temperature changes, electromagnetic interference, or tilting, can affect a magnetic compass allows individuals to make informed decisions in challenging situations.\nProper Equipment Usage: # Users of magnetic compasses need to be aware of potential disturbances to ensure proper usage and accurate readings. This knowledge is significant for individuals who rely on compasses in their professions, such as pilots, mariners, hikers, and military personnel.\nCalibration and Correction: # Awareness of potential disturbances prompts individuals to regularly calibrate their compasses and apply corrections when necessary. Calibrating the compass compensates for deviations introduced by external factors, maintaining the instrument\u0026rsquo;s accuracy.\nEducational Purposes: # Understanding how a magnetic compass works and how it can be disturbed is a fundamental aspect of education in navigation and orienteering. It allows individuals to develop skills in using a compass effectively and helps them troubleshoot when faced with challenges.\nTechnology Development: # For engineers and designers involved in developing magnetic compasses or navigation systems, understanding potential disturbances is essential for creating robust and reliable instruments. This knowledge informs the design of compensation mechanisms and technologies that enhance accuracy.\nUnderstanding how a magnetic compass can be disturbed is crucial for ensuring accurate and reliable navigation, promoting safety, and enabling the effective use of this essential tool in various contexts. Users and professionals benefit from this understanding to mitigate potential errors and enhance the performance of magnetic compasses.\nWhat are the disturbance variables? # Disturbance variables in a compass refer to external factors or influences that can affect the accuracy of the compass reading. These disturbances can lead to errors in the measurement of magnetic direction. Some standard disturbance variables in a compass include:\nMagnetic Declination: # The difference between true north and magnetic north. Compass readings are affected by this declination and must be considered for accurate navigation.\nMagnetic declination, often referred to as \u0026ldquo;declination,\u0026rdquo; is the angular difference between true north (geographic north) and magnetic north (the direction a magnetic compass points). It is a crucial factor in navigation, helping correct the difference between magnetic and true north when using a magnetic compass.\nThe Earth\u0026rsquo;s magnetic field is not aligned perfectly with its geographic axis, which causes the magnetic north pole and the geographic (true) north pole to be located at different points on the Earth\u0026rsquo;s surface. As a result, when you use a magnetic compass, it points not directly toward the North Pole but to the Magnetic North Pole, creating a discrepancy between the Magnetic North and True North.\nMagnetic declination is expressed in degrees and is specified with three critical pieces of information:\nValue : The angular measurement in degrees of the angle between true north and magnetic north.\nDirection : Whether magnetic north is east or west of true north. If the declination is east, it is expressed as a positive value; if it is west, it is described as a negative value.\nYear of Measurement : Magnetic declination is not constant and changes over time due to Earth\u0026rsquo;s magnetic field variations. Therefore, it\u0026rsquo;s essential to know the year for which the declination value is accurate.\nFor example, a magnetic declination of +10 degrees means that magnetic north is 10 degrees east of true north at the time of measurement. If the declination is -5 degrees, magnetic north is 5 degrees west of true north.\nNavigational charts and maps often include information about the local magnetic declination, and compass users need to account for this when navigating to determine the actual direction accurately. It\u0026rsquo;s crucial to adjust for magnetic declination to obtain accurate readings by adding or subtracting the declination value from the compass reading, depending on whether it is east or west.\nHow do we extract the magnetic declination from the field? # Depending on the available resources, you can use various methods to extract the magnetic declination in the field. Here are a few common approaches:\nUse Online Tools or Apps : Many online tools and mobile apps provide real-time magnetic declination information based on location. GPS-enabled apps can determine your geographical coordinates and provide the magnetic declination for that specific location.\nConsult Topographic Maps : Topographic maps often include information about magnetic declination for specific regions. Look for a declination diagram or note on the map. Remember that the declination information may have an associated year, so you may need to adjust for changes over time.\nUse a Compass with Adjustable Declination : Some compasses feature an adjustable setting. You can manually set the compass to the local magnetic declination for your specific location. Check the user manual for your compass to learn how to adjust the declination.\nGovernmental or Geological Surveys : Governmental or geological survey agencies often provide magnetic declination information for various locations. Some countries have online databases or publications that offer this information.\nRefer to Magnetic Declination Tables : Magnetic declination tables are available in navigation books, manuals, or online resources. These tables provide declination values for specific locations and years. Find the relevant information based on your current or planned location.\nUse the Sun and Shadow Method : If you don\u0026rsquo;t have access to tools or maps, you can estimate the cardinal directions by observing the sun\u0026rsquo;s position in the sky and using shadow lengths. This method provides a rough indication of the direction and can be combined with magnetic declination information from other sources.\nConsider the year associated with the magnetic declination information as it changes over time. If you cannot find current information for your location, you can estimate the magnetic declination based on the general trend in your region and adjust your compass readings accordingly. Always use multiple sources and methods for increased accuracy.\nHow do we Use the Sun and Shadow Method to define local magnetic declination? # The Sun and Shadow Method can estimate the local magnetic declination. This method involves observing the sun\u0026rsquo;s position in the sky and the length of shadows. Here\u0026rsquo;s a step-by-step guide:\nMaterials Needed:\nStick or Dowel : A straight stick or dowel placed vertically in the ground.\nFlat Surface : A level surface where the stick\u0026rsquo;s shadow can be easily observed.\nSteps:\nSetup :\nPlace the stick or dowel vertically on the ground on a flat surface. Ensure that the stick is stable and straight.\nMark the Tip of the Shadow :\nWait for a sunny day and observe the shadow cast by the stick. Mark the tip of the shadow with a small object or draw a line.\nWait for Some Time :\nAllow some time (30 minutes to an hour), ensuring the sun moves across the sky.\nMark the New Tip of the Shadow :\nOnce again, mark the tip of the shadow in the same manner as before.\nConnect the Marks :\nDraw a line connecting the two marks on the ground. This line represents the west-east direction.\nDetermine True North :\nUse a compass to determine the magnetic north direction. Align the compass needle with the north-south line on the compass.\nCompare the Lines :\nCompare the direction of the magnetic north indicated by the compass with the west-east line drawn from the shadow method.\nCalculate the Declination :\nThe angle between the two lines represents the local magnetic declination. If the magnetic north indicated by the compass is east of the drawn line, the declination is positive; if it\u0026rsquo;s west, it is negative.\nRemember, this method provides only an estimate and may not be as accurate as using precise instruments or online resources.\nCross-reference : Cross-reference your results with known magnetic declination values for your location if possible. Additionally, this method is most accurate during sunrise or sunset when the sun is closer to the horizon.\nMagnetic Deviation: # Local magnetic fields, such as those created by nearby metal objects or electronic devices, can cause the compass needle\u0026rsquo;s orientation deviations.\nMagnetic deviation refers to the error introduced in the reading of a magnetic compass due to the influence of local magnetic fields within the immediate vicinity of the compass. Unlike magnetic declination, a global or regional phenomenon, the magnetic deviation is specific to the conditions around the compass at a given location.\nVarious factors can contribute to magnetic deviation, and they often result from the presence of nearby metallic objects or electronic equipment that can create their own magnetic fields. The Earth\u0026rsquo;s magnetic field interacts with these local fields, causing the compass needle to deviate from its proper magnetic heading.\nTo correct for magnetic deviation and obtain accurate compass readings, mariners and navigators typically perform what is known as compass compensation or calibration. This involves determining the specific magnetic deviation at a particular location and then applying corrections to subsequent compass readings.\nThe process of compensating for magnetic deviation may involve adjusting the position of the compass, using compensating magnets, or employing other methods to counteract the local magnetic influences. This calibration is essential for accurate navigation, especially in maritime and aviation contexts where precise direction is crucial.\nIt\u0026rsquo;s worth noting that magnetic deviation is location-specific and can change as the compass moves to different areas with varying magnetic influences. Therefore, navigators must know the local magnetic deviation and adjust their compass readings accordingly for accurate and reliable navigation.\nElectromagnetic Interference (EMI): # Electromagnetic Interference (EMI) refers to the disruption or degradation of the performance of an electronic device caused by electromagnetic signals from external sources. In the context of a magnetic compass, EMI can introduce disturbances that affect the accuracy of compass readings.\nThe disturbance caused by EMI on a magnetic compass arises from the fact that the compass needle is sensitive to magnetic fields, and any external electromagnetic fields can interfere with its proper functioning. Here are some ways in which EMI can disturb a magnetic compass:\nElectronic Devices : Proximity to electronic devices that emit electromagnetic radiation, such as radios, motors, radar equipment, or other electronic instruments, can introduce interference. The electromagnetic fields produced by these devices may influence the magnetic field sensed by the compass, leading to inaccurate readings.\nPower Lines : Strong electromagnetic fields generated by high-voltage power lines can affect the magnetic field around the compass. Navigators and pilots must be cautious when using a compass close to power lines to avoid interference.\nVehicle Electronics : Electrical systems and various electronic components can generate electromagnetic fields in vehicles. Placing a magnetic compass close to such components can affect the compass\u0026rsquo;s accuracy.\nMagnets and Magnetic Materials : Strong magnets or magnetic materials near a compass can disrupt the Earth\u0026rsquo;s magnetic field, leading to the compass needle\u0026rsquo;s orientation deviations.\nTo minimise the impact of EMI on a magnetic compass, it\u0026rsquo;s essential to follow best practices, including:\nMaintain Distance : Keep the compass away from electronic devices, power sources, and other equipment that may produce electromagnetic fields.\nCalibration : Periodically calibrate the compass to account for any deviations introduced by EMI. This is especially important in environments where electronic equipment is in use.\nShielding : Some compasses are designed with shielding to reduce the effects of external electromagnetic interference. Using such compasses can be beneficial in environments with high EMI.\nIn critical applications such as aviation and marine navigation, where accurate compass readings are essential for safety, it\u0026rsquo;s important to be aware of potential EMI sources and take measures to mitigate their impact on the magnetic compass.\nFerrous (Iron) Objects: # Ferrous (iron-containing) objects can disturb a magnetic compass due to their influence on the local magnetic field. The Earth\u0026rsquo;s magnetic field interacts with ferrous materials, and when a magnetic compass is close to such objects, it can lead to errors in the compass readings. Here\u0026rsquo;s how ferrous objects can disturb a magnetic compass:\nMagnetic Attraction : Ferrous objects are attracted to magnets and can become magnetised themselves. When a compass is brought near a ferrous object, the object can influence the magnetic needle of the compass, causing it to deflect toward the ferrous material.\nLocal Magnetic Fields : Ferrous objects can create their own local magnetic fields, which may differ from the Earth\u0026rsquo;s. These additional magnetic fields can interfere with the proper alignment of the compass needle, leading to inaccurate readings.\nDistortion of Earth\u0026rsquo;s Magnetic Field : Large ferrous objects, such as steel structures, vehicles, or metal equipment, can distort the Earth\u0026rsquo;s magnetic field in their vicinity. This distortion affects the behaviour of the compass needle, causing it to deviate from its true magnetic heading.\nTo minimise the impact of ferrous objects on a magnetic compass, consider the following:\nKeep a Safe Distance : Maintain a sufficient distance between the magnetic compass and ferrous objects. This is especially important in environments where large metal structures or equipment are present.\nCalibration : Periodically calibrate the compass to account for any deviations introduced by the presence of ferrous objects. This is particularly important when navigating in areas where such objects are common.\nAwareness : Be aware of the surroundings and potential sources of magnetic interference. This awareness allows navigators to anticipate and correct any disturbances caused by ferrous materials.\nProximity to Magnets: # Proximity to magnets can disturb a magnetic compass due to the influence of the magnetic field created by the magnets. A magnetic compass operates based on the Earth\u0026rsquo;s magnetic field, and any additional magnetic fields introduced by nearby magnets can interfere with the proper functioning of the compass. Here\u0026rsquo;s how proximity to magnets can disturb a magnetic compass:\nMagnetic Attraction : Magnets attract materials containing iron and other ferrous metals. If a compass is brought close to a strong magnet, the magnet may attract the ferrous components within the compass itself, leading to mechanical disturbances in the compass needle\u0026rsquo;s movement.\nMagnetic Fields : Magnets produce magnetic fields, and these fields can affect the delicate balance of forces acting on the compass needle. The compass needle aligns with the Earth\u0026rsquo;s magnetic field, but a strong external magnetic field from a nearby magnet can cause the needle to deviate from its standard orientation.\nMagnetisation : Over time, exposure to a strong magnetic field can cause the materials within the compass to become magnetised. This can result in a persistent deviation of the compass needle even after the magnet is no longer nearby.\nTo minimise the impact of proximity to magnets on a magnetic compass:\nMaintain Distance : Keep the compass away from strong magnets. This is especially important when powerful magnets are used, such as in magnetic tools or equipment.\nCalibration : If a compass has been exposed to magnets, it may need to be recalibrated to ensure accurate readings. Calibration involves compensating for any deviations caused by the magnet\u0026rsquo;s influence.\nAwareness : Be mindful of the presence of magnets near a compass. In some situations, such as in laboratories or workshops where magnets are commonly used, navigators and users of compasses should exercise caution.\nTilting and Inclination: # Tilting and inclination can disturb a magnetic compass due to the effect of gravity on the compass needle. A magnetic compass relies on a balanced interplay of magnetic forces and the force of gravity to align itself with the Earth\u0026rsquo;s magnetic field. Here\u0026rsquo;s how tilting and inclination can affect a magnetic compass:\nGravity and Compass Needle Balance : The compass needle is a magnet, and it is typically mounted on a pivot, allowing it to rotate freely. The needle aligns with the Earth\u0026rsquo;s magnetic field, with the north-seeking end pointing toward the magnetic north. The force of gravity acts on the needle, attempting to pull it downward.\nTilting the Compass : When the compass is tilted or held at an angle, the force of gravity may no longer act directly along the axis of the needle. Instead, gravity introduces a component that pulls the needle toward the lower end of the compass housing. This can cause the needle to tilt and lead to errors in the compass reading.\nInclination : In addition to tilting, inclination refers to the angle between the magnetic field lines and the horizontal plane. As you move toward the magnetic poles, the inclination increases. Tilting a compass in regions with high inclination can result in more significant errors, as the force of gravity significantly impacts the compass needle.\nTo minimise the impact of tilting and inclination on a magnetic compass:\nHold the Compass Level : To obtain accurate readings, users should hold the compass as levelly as possible. Keeping the compass level allows the needle to respond primarily to the Earth\u0026rsquo;s magnetic field and minimises the influence of gravity-induced tilting.\nCompensation : Some compasses are designed with built-in features to compensate for the effects of tilting and inclination. These features may include liquid-filled capsules or gimbaled systems that allow the compass to remain level even if the user is on an incline.\nCalibration : Periodically calibrate the compass to ensure accurate readings, especially if it has been subjected to conditions that may affect its performance.\nTemperature Variations: # Temperature variations can disturb a magnetic compass due to their impact on the compass\u0026rsquo;s materials and components, affecting the compass needle\u0026rsquo;s alignment. The Earth\u0026rsquo;s magnetic field interacts with the materials within the compass, and temperature changes can influence the magnetic properties of these materials. Here\u0026rsquo;s how temperature variations can affect a magnetic compass:\nExpansion and Contraction of Materials : Different materials used in constructing a compass can expand or contract with temperature changes. This includes the magnetised needle, the fluid (if the compass is liquid-filled), and other components. These changes in dimension can lead to mechanical stress and alter the alignment of the compass needle.\nMagnetic Properties of Materials : The magnetic properties of materials, including the magnetic needle itself, can be temperature-dependent. Temperature variations can affect the strength of the magnetic field generated by the compass needle, leading to changes in its behaviour.\nFluid in Liquid-Filled Compasses : Some compasses are filled with a liquid (often oil) to dampen needle oscillations and improve stability. Temperature changes can cause the fluid to expand or contract, affecting the buoyancy and damping characteristics. This, in turn, can influence the movement of the compass needle.\nTo minimise the impact of temperature variations on a magnetic compass:\nCalibration : Periodically calibrate the compass to account for any deviations introduced by temperature changes. Calibration involves adjusting the compass to ensure accurate readings under specific conditions.\nTemperature Compensation : Some advanced compasses are designed with temperature compensation features. These features aim to minimise the influence of temperature on the compass needle by incorporating materials and mechanisms that respond predictably to temperature changes.\nStorage Conditions : When not in use, storing the compass in an environment with a stable temperature is advisable. Extreme temperature fluctuations and exceptionally rapid changes can introduce stress on the compass components.\nUsers should be aware of the conditions under which the compass is used and take appropriate measures to compensate for any deviations caused by temperature changes.\nExternal Forces: # External forces can disturb a magnetic compass by exerting physical influences on the compass needle, leading to deviations from its proper alignment. The compass needle is a sensitive component that responds to magnetic and mechanical forces. Here are some ways in which external forces can affect a magnetic compass:\nVibration and Movement : External forces, such as vibrations from machinery, vehicle motion, or other forms of movement, can cause the compass needle to oscillate or vibrate. This can lead to temporary fluctuations in the compass reading, making obtaining a stable and accurate direction challenging.\nMechanical Shock : Sudden jolts or impacts, as might occur during rough handling or if the compass is dropped, can introduce mechanical stress on the compass needle. This stress may result in a temporary or permanent deviation from the correct alignment.\nWind and Air Flow : In outdoor environments, wind and airflow can exert forces on the compass needle, causing it to deflect. This effect is more pronounced when the compass is held in the hand or mounted on a surface exposed to wind.\nWater Flow (Marine Applications) : In marine navigation, water flow around the hull of a vessel can create turbulence and affect the compass reading. This is known as \u0026ldquo;compass deviation\u0026rdquo;, caused by the vessel\u0026rsquo;s movement through the Earth\u0026rsquo;s magnetic field.\nTo minimise the impact of external forces on a magnetic compass:\nStabilise the Compass : Hold the compass steady to minimise vibrations and movements. This is especially important when taking readings for navigation or orientation.\nDampening Mechanisms : Some compasses have dampening mechanisms to minimise needle oscillations. These mechanisms help stabilise the needle and reduce the impact of external forces.\nUse in Stable Conditions : Use the compass in conditions with minimal external forces. Avoid using the compass near machinery, strong air currents, or turbulent water flow that could introduce disturbances.\nRegular Calibration : Periodically calibrate the compass to correct for any deviations introduced by external forces or mechanical stress.\nBeing mindful of external forces and stabilising the compass during use is essential for obtaining accurate readings, especially in navigation contexts where precision is crucial.\nTo minimise the impact of these disturbance variables, it\u0026rsquo;s essential to use the compass correctly, away from sources of interference, and to periodically calibrate the compass based on the specific conditions of use. Modern electronic compasses may also have built-in features to compensate for some of these disturbances.\nHow do I identify in the field that my magnetic compass is disturbed? # Identifying that your magnetic compass is disturbed in the field ensures accurate navigation. Here are some signs that may indicate disturbances to your magnetic compass:\nInconsistent Readings : If you notice that the compass readings fluctuate or vary significantly without any apparent reason, it could be a sign of disturbances.\nNeedle Oscillation : The compass needle may oscillate or vibrate, especially when exposed to external forces like wind or vibrations. This can make it challenging to obtain a stable reading.\nInaccurate Direction : If your observed direction does not align with known landmarks or your expected course, it may indicate disturbances affecting the compass\u0026rsquo;s accuracy.\nUnusual Needle Behavior : Any erratic behaviour of the compass needle, such as sudden jumps or swings, may suggest external influences on the magnetic field.\nFailure to Settle : After coming to a stop, the compass needle should settle and point consistently in one direction. Distances may be at play if it continues to move or doesn\u0026rsquo;t settle.\nInfluence of Nearby Objects : Ferrous objects, magnets, or electronic devices near the compass may cause deviations. Be aware of your surroundings and potential sources of interference.\nSuspected Interference Sources : If you are in an environment with known sources of electromagnetic interference (EMI), such as electronic equipment or power lines, be vigilant for compass disturbances.\nTo confirm if your magnetic compass is disturbed: # Test in Different Locations : Move away from potential interference sources to a different location and observe if the compass behaviour changes. This helps rule out local disturbances.\nCalibration Check : Periodically calibrate your compass to account for any deviations caused by external factors. Disturbances may be present if the calibration consistently fails to provide accurate readings.\nUse Multiple Navigation Aids : Cross-reference your magnetic compass readings with other navigation aids, such as a GPS or map and landmarks, to verify accuracy.\nWhat are the possible corrective actions? # If you suspect disturbances, take corrective actions:\nHold the Compass Level : Keep the compass level to minimise the impact of tilting and inclination.\nCalibrate : Calibrate the compass according to the manufacturer\u0026rsquo;s instructions.\nMove Away from Disturbances : Distance yourself from known or suspected sources of interference.\nBeing vigilant and proactive in identifying and addressing disturbances will help ensure the reliability of your magnetic compass in the field.\nConclusion: # We discussed various aspects of magnetic compasses and the factors that can disturb their accuracy. We covered concepts such as magnetic declination, the angle between true north and magnetic north, and how it varies by location. Disturbances to a magnetic compass can arise from factors like ferrous objects, electromagnetic interference, temperature variations, and external forces.\nWe explored the impact of tilting, inclination, and proximity to magnets on compass readings. Understanding these disturbances is crucial for accurate navigation. We also highlighted the importance of recognising signs of disorder in the field, such as inconsistent readings or needle oscillation, and provided tips for calibration.\nWe emphasised the significance of identifying and mitigating disturbances to ensure the reliability of a magnetic compass. We discussed practical methods for extracting magnetic declination in the field, including online tools, topographic maps, and compass adjustments. The Sun and Shadow Method was explained as a simple technique for estimating local magnetic declination.\nOverall, we underscored the importance of being aware of potential disturbances, calibrating compasses regularly, and employing various methods to obtain accurate readings in different environments and scenarios.\nHappy Navigating\nSven\n","date":"20 December 2023","externalUrl":null,"permalink":"/posts/precision-in-the-wilderness-how-to-navigate-safely-despite-magnetic-compass-disturbances/","section":"Posts","summary":"Understanding the disturbance of a magnetic compass is crucial for accurate navigation. External factors like magnetic declination, ferrous objects, electromagnetic interference, temperature variations, and external forces can affect compass readings. Magnetic declination represents the angular difference between true north and magnetic north. Ferrous objects and nearby magnets can alter the local magnetic field, causing deviations. Electromagnetic interference from electronic devices may disrupt compass accuracy. Temperature variations influence materials and magnetic properties within the compass. External forces, such as vibrations and movement, can lead to temporary disturbances. To maintain accuracy, users must know these factors, hold the compass level, calibrate periodically, and avoid interference sources. Understanding and mitigating compass disturbances are essential for reliable and precise direction finding in navigation and safety-critical situations. Let’s have a look at these topics.\n","title":"Precision in the Wilderness: How to Navigate Safely Despite Magnetic Compass Disturbances","type":"posts"},{"content":"Embark on a journey to understand the Universal Transverse Mercator (UTM) coordinate system—a foundational tool in mapping and navigation. Unveil the historical evolution that led to its development during World War II, explore the technical intricacies that make it a global standard, and discover the practical applications that range from topographic mapping to field navigation.\nThis exploration into the UTM system promises a comprehensive view of its role in shaping accurate representations of the Earth\u0026rsquo;s surface, providing a standardised language for geographic information. Delve into its relevance in diverse fields, uncover the nuances of its implementation, and grasp the practical steps to harness its power in real-world scenarios.\nWenn Du den Artikel auf Deutsch lesen möchtest..\nhttps://outdoorskills.blog/2023/12/19/mapping-mastery-entschlusselung-der-globalen-auswirkungen-von-utm-koordinaten/\nWhether you\u0026rsquo;re a cartography enthusiast, a GIS professional, or simply curious about the intricacies of spatial mapping, this exploration into the UTM coordinate system invites you to unravel the layers of its history, technical specifications, and practical applications. Gain insights that transcend geographic boundaries and discover UTM\u0026rsquo;s essential role in the precision and consistency of modern mapping endeavours.\nThe History of UTM Early Mapping Systems: Military Needs in World War II: Army Map Service (AMS): Transverse Mercator Projection: UTM Adoption: International Standardization: UTM in Civilian Applications: Global Coverage: The Technical Specification of UTM Map Projection: Zone Division: Central Meridian: False Easting and False Northing: Coordinate Units: Datum: Scale Factor: Coordinate Ranges: What is the connection between the UTM and the WGS84 system? WGS84: UTM (Universal Transverse Mercator): Connection between UTM and WGS84: What other Coordinate Systems are based on Transverse Mercator projection as well? State Plane Coordinate System (SPCS): British National Grid (BNG): Irish Grid: UTM relevance for topographic hiking maps? Global Standardization: Local Accuracy: Ease of Use: UTM Grid Overlay: Integration with GPS: Suitability for Regional Mapping: Cartographic Conventions: UTM usability in polar regions Excessive Distortion: Coordinate Range Limitations: Crossing UTM Zones: Polar Stereographic Projection: Geographic Coordinates (Latitude and Longitude): Practical Usage of UTM Conclusion: The History of UTM # The Universal Transverse Mercator (UTM) coordinate system has its roots in the efforts to create a global mapping and coordinate system that would facilitate accurate and consistent mapping of the Earth\u0026rsquo;s surface. Here is a brief history of the development of the UTM system:\nEarly Mapping Systems: # Before the UTM system, various map projections and coordinate systems were used to map different regions, leading to a lack of consistency and interoperability on a global scale.\nMilitary Needs in World War II: # The development of UTM can be traced back to the military needs during World War II. The U.S. Army found that existing coordinate systems were unsuited for large-scale military operations that covered multiple map sheets and required accurate distance and direction measurements.\nArmy Map Service (AMS): # - The UTM system was developed by the U.S. Army Map Service (AMS) in the 1940s. The AMS worked on creating a coordinate system that would be easy to use, eliminate distortions in local mapping, and provide accurate measurements for military purposes.\nTransverse Mercator Projection: # The UTM system is based on the transverse Mercator map projection, a cylindrical projection that reduces distortion to a narrow longitudinal extent. Each UTM zone spans 6 degrees of longitude.\nUTM Adoption: # The military and civilian mapping agencies adopted the UTM system for its advantages in local mapping and its ability to provide a global framework.\nInternational Standardization: # The UTM system gained international recognition, and its use was standardized. The International Map of the World adopted the UTM system, contributing to its widespread acceptance.\nUTM in Civilian Applications: # As technology advanced and civilian applications for mapping and navigation grew, UTM became widely used in GIS (Geographic Information Systems), cartography, and GPS (Global Positioning System) applications.\nGlobal Coverage: # The UTM system divides the Earth\u0026rsquo;s surface into zones, each with its coordinate system. Collectively, these zones cover the entire globe, providing a consistent and efficient method for representing locations.\nToday, the UTM system is a standard for mapping and navigation, and it is widely used in various fields such as surveying, engineering, and geographic information systems. Using UTM coordinates referenced to the WGS84 datum ensures global interoperability and accuracy in spatial data representation.\nThe Technical Specification of UTM # The Universal Transverse Mercator (UTM) coordinate system has specific technical specifications that define its parameters and characteristics. Here are the key technical specifications of the UTM system:\nMap Projection: # UTM is based on the Transverse Mercator map projection. The Transverse Mercator projection is a cylindrical tangent along a meridian (line of longitude).\nZone Division: # The Earth is divided into longitudinal zones, each spanning 6 degrees longitude. There are 60 UTM zones, numbered consecutively from 1 to 60, starting from 180 degrees west.\nCentral Meridian: # Each UTM zone has a central meridian along which there is no distortion. The central meridian for each zone is at the centre of the 6-degree longitudinal extent of the zone.\nFalse Easting and False Northing: # To ensure that all coordinates in a UTM zone are positive, a false easting value of 500,000 meters is added to all x-coordinates (easting values). The false northing for the northern hemisphere is 0 at the equator; for the southern hemisphere, it is 10,000,000 meters.\nCoordinate Units: # UTM coordinates are typically expressed in meters. The easting values are measured from the central meridian, and the northing values are measured from the equator or the false northing value.\nDatum: # UTM coordinates are often referenced to a specific geodetic datum. WGS84 (World Geodetic System 1984) is commonly used as the datum for UTM coordinates, ensuring global interoperability with GPS and other positioning systems.\nScale Factor: # The scale factor is the ratio of the scale along the central meridian to the scale at the equator. This scale factor is kept within certain limits in the UTM system to minimise distortion.\nCoordinate Ranges: # UTM coordinates are limited within specific ranges to ensure accuracy and avoid ambiguity. The easting values typically range from 166,021 meters to 833,021 meters, and northing values range from 0 meters at the equator to 10,000,000 meters at the poles.\nThese technical specifications ensure the UTM system provides an accurate and consistent framework for mapping and navigation over the Earth\u0026rsquo;s surface. Users should know the specific parameters associated with the UTM zone they are working in to interpret and use UTM coordinates accurately.\nWhat is the connection between the UTM and the WGS84 system? # UTM (Universal Transverse Mercator) and WGS84 (World Geodetic System 1984) are related to geospatial information and mapping but serve different purposes.\nWGS84: # WGS84 is a geodetic datum, a reference system for specifying locations on the Earth\u0026rsquo;s surface. It provides a standard framework for measuring positions, distances, and elevations. WGS84 is a reference for GPS (Global Positioning System) and is widely adopted as the standard for mapping and navigation.\nUTM (Universal Transverse Mercator): # UTM is a map projection and coordinate system that divides the world into a series of zones, each with its coordinate system. The UTM projection is based on a cylindrical system where the Earth\u0026rsquo;s surface is divided into transverse Mercator projections. Each UTM zone is defined by its central meridian, and its coordinates are measured in meters east and north from that central meridian and the equator. UTM coordinates are often expressed in meters, providing a Cartesian coordinate system suitable for local and regional mapping.\nConnection between UTM and WGS84: # UTM coordinates are often referenced to the WGS84 datum. This means the coordinates provided in the UTM system are ultimately based on the WGS84 geodetic datum. WGS84 provides the underlying reference framework for the geographic coordinates used in UTM projections. UTM zones are designed to minimise distortions within each zone while still being based on a global geodetic framework like WGS84. When using GPS devices or GIS (Geographic Information System) software that uses UTM coordinates, it is common for the GPS data to be collected and stored in the WGS84 datum.\nIn summary, while WGS84 is a geodetic datum providing a reference for global positioning, UTM is a coordinate system and map projection designed for specific regions, and it is often used with coordinates referenced to the WGS84 datum for consistency in global mapping systems.\nWhat other Coordinate Systems are based on Transverse Mercator projection as well? # The Universal Transverse Mercator (UTM) coordinate system is itself a coordinate system based on the Transverse Mercator map projection. It is designed to provide accurate and consistent representations of locations on the Earth\u0026rsquo;s surface within specific zones. Each UTM zone is a separate coordinate system , and the entire UTM system covers the globe by dividing it into a series of zones.\nIn addition to the UTM coordinate system, other coordinate systems are based on the Transverse Mercator projection, but they may not be synonymous with UTM. Here are a few examples:\nState Plane Coordinate System (SPCS): # The State Plane Coordinate System is used in the United States to map large regions, such as individual states or groups. It utilises a Transverse Mercator projection and can have various coordinate zones.\nBritish National Grid (BNG): # The British National Grid is based on the Ordnance Survey National Grid in the United Kingdom. It employs the Transverse Mercator projection and is divided into grid squares. While it is similar to UTM, it is specific to the United Kingdom.\nIrish Grid: # The Irish Grid is used in Ireland and is based on the Transverse Mercator projection. It is similar to the British National Grid but tailored to the Irish mapping system.\nIt\u0026rsquo;s important to note that while these coordinate systems use the Transverse Mercator projection, they may have different parameters, origins, and zone divisions compared to UTM. UTM is globally standardised and designed to cover the entire Earth by dividing it into 6-degree longitudinal zones.\nIn summary, UTM is a specific implementation of the Transverse Mercator projection that provides a standardised global coordinate system. Other coordinate systems, like the State Plane Coordinate System, British National Grid, and Irish Grid, are based on the same projection but are designed for more localised applications.\nUTM relevance for topographic hiking maps? # The Universal Transverse Mercator (UTM) coordinate system is highly relevant for topographic maps. It is one of the most commonly used coordinate systems for topographic mapping. Here\u0026rsquo;s why UTM is appropriate in the context of topographic maps:\nGlobal Standardization: # UTM provides a globally standardised coordinate system, making integrating and sharing topographic information across different regions and countries easy. This standardisation facilitates interoperability in mapping and navigation.\nLocal Accuracy: # UTM minimises distortion within each 6-degree longitudinal zone, providing accurate representations of local areas. This is crucial for topographic maps, where precise measurements and representations of terrain features are essential.\nEase of Use: # UTM coordinates are expressed in meters, which simplifies calculations and measurements. This makes it convenient for surveyors, cartographers, and other professionals to create and use topographic maps.\nUTM Grid Overlay: # UTM zones are often overlaid on topographic maps, creating a grid system that facilitates easy identification of locations and distances. This grid is handy for field navigation and measurement.\nIntegration with GPS: # Many GPS devices and mapping software use UTM coordinates. Since UTM is often referenced to the WGS84 datum, commonly used by GPS systems, it allows for seamless integration of GPS data with topographic maps.\nSuitability for Regional Mapping: # UTM is designed to minimise distortion within each zone, making it well-suited for regional mapping. Topographic maps typically cover specific regions, and UTM\u0026rsquo;s zoning system aligns with this approach.\nCartographic Conventions: # Many national mapping agencies adopt UTM as the coordinate system for their topographic maps. This consistency in choice simplifies the production and use of maps.\nWhile UTM is widely used, it\u0026rsquo;s important to note that local coordinate systems may also be employed in some cases for topographic mapping, especially in regions with specific mapping conventions. However, UTM is a common and practical choice for topographic maps for a global or widely applicable standard.\nUTM usability in polar regions # The Universal Transverse Mercator (UTM) coordinate system is unsuited for polar regions. UTM is based on the Transverse Mercator projection, a cylindrical projection that becomes highly distorted near the poles. The distortion increases as you move away from the central meridian of the UTM zone.\nSpecific issues with using UTM in the polar regions include:\nExcessive Distortion: # The UTM projection is distorted near the poles, making it unsuitable for accurate mapping in these areas. Distortion increases as you approach the pole, and it becomes infinite at the pole itself.\nCoordinate Range Limitations: # The UTM coordinate system has defined ranges for easting and northing values, which are exceeded near the poles. This limitation makes it impractical to use UTM for mapping in polar regions.\nCrossing UTM Zones: # A given location may fall into multiple UTM zones at high latitudes, which could complicate coordinate representation. The UTM system is designed to map specific longitudinal zones accurately, and the transition between zones can introduce additional challenges.\nFor mapping in polar regions, other coordinate systems are typically used. Some common alternatives include:\nPolar Stereographic Projection: # The Polar Stereographic projection is often employed for mapping near the poles. It minimises distortion in polar regions and is suitable for navigation and mapping in high latitudes.\nGeographic Coordinates (Latitude and Longitude): # Geographic coordinates (latitude and longitude) are universally applicable and do not suffer from distortion issues near the poles. However, they can pose challenges in terms of measuring distances accurately.\nWhen working in polar regions, it\u0026rsquo;s crucial to choose a coordinate system and map projection specifically designed to handle the unique challenges of those areas. The choice may depend on the specific requirements of the mapping project and the desired balance between accuracy and distortion.\nPractical Usage of UTM # Using the Universal Transverse Mercator (UTM) coordinate system practically involves understanding its key concepts and employing the coordinates in mapping, navigation, surveying, or other related activities. Here\u0026rsquo;s a step-by-step guide on how to use UTM practically:\nIdentify the UTM Zone :\nDetermine the UTM zone for the area of interest. 6-degree longitudinal segments define UTM zones, each with its coordinate system.\nObtain UTM Coordinates :\nObtain the UTM coordinates for specific locations within the chosen UTM zone. Coordinates consist of an easting value (measured in meters eastward from the central meridian) and a northing value (measured northward from the equator or a false northing value).\nMap Overlay :\nIf working with a map, overlay the UTM grid on the map. Many topographic maps and mapping software include UTM grid lines, making identifying locations and measuring distances easy.\nCoordinate Conversion :\nIf you have coordinates in a different coordinate system (e.g., latitude and longitude), you may need to convert them to UTM coordinates. This conversion can be done using specialised software, online tools, or manual calculations.\nField Navigation :\nIn the field, use UTM coordinates for navigation. Many GPS devices allow you to switch between coordinate systems, and selecting UTM can provide accurate position information.\nSurveying and Mapping :\nWhen conducting surveys or creating maps, use UTM coordinates to represent features and measurements accurately. Ensure that your surveying equipment or mapping software uses the correct UTM zone and datum.\nAccount for Datum :\nBe aware of the geodetic datum associated with the UTM coordinates. WGS84 is a commonly used datum, but regional datums may also be applicable. Ensure consistency between the datum used for data collection and mapping or analysis.\nSoftware Tools :\nUtilise GIS software or other mapping tools that support UTM coordinates. These tools often provide functionalities for measuring distances, calculating areas, and performing various spatial analyses based on UTM coordinates.\nCoordinate Limitations :\nBe mindful of the limitations of UTM coordinates, especially near zone boundaries or in high-latitude regions where distortion increases. Understand the coordinate ranges and how they may impact your work.\nDocumentation :\nDocument the coordinate system and datum used for your data. This information is crucial for data sharing and ensures that others can correctly interpret and use your spatial data.\nBy following these steps and considering the practical aspects of UTM, you can effectively use this coordinate system in various applications related to mapping, navigation, and geospatial analysis.\nConclusion: # The Universal Transverse Mercator (UTM) coordinate system is a global standard based on the Transverse Mercator map projection. Developed initially for military purposes during World War II, UTM divides the Earth into 6-degree longitudinal zones, each with its coordinate system. It minimizes distortion within each zone, providing accuracy for local mapping. UTM coordinates are commonly referenced to the WGS84 datum and expressed in meters, making them convenient for various applications, including topographic maps. It should be noted that Universal Transverse Mercator (UTM) projection may not be appropriate for use in polar regions as it can cause distortion issues. The practical application of this projection method involves several steps, such as identifying the correct UTM zone, obtaining the coordinates, overlaying maps, converting coordinates if necessary, navigating in the field and utilising GIS software for mapping and analysis. It is crucial to consider the coordinate limitations while documenting the chosen datum.\n","date":"19 December 2023","externalUrl":null,"permalink":"/posts/mapping-mastery-decoding-the-global-impact-of-utm-coordinates/","section":"Posts","summary":"Embark on a journey to understand the Universal Transverse Mercator (UTM) coordinate system—a foundational tool in mapping and navigation. Unveil the historical evolution that led to its development during World War II, explore the technical intricacies that make it a global standard, and discover the practical applications that range from topographic mapping to field navigation.\n","title":"Mapping Mastery: Decoding the Global Impact of UTM Coordinates","type":"posts"},{"content":"The World Geodetic System 1984 (WGS 84) is a geodetic reference system used to describe the shape and size of the Earth. We will delve into its historical context, technical specifications, global significance, and practical applications.\nIntroduction: Understanding Geodetic Reference Systems Historical Evolution of Geodetic Reference Systems WGS 84 Basic Technical Specifications Satellite Contributions to WGS 84 WGS 84 and Global Positioning System (GPS) Practical Applications of WGS 84 Challenges and Evolving Standards Conclusion: WGS 84 in a Global Context How to use WGS84? Understand WGS 84 Parameters: Coordinate Representation: Use GPS Devices: Mapping and GIS Applications: Aviation and Maritime Navigation: Surveying and Geodetic Measurements: Conversion Tools: Online Mapping Services: Programming and APIs: Stay Informed about Updates: Collaboration with Others: How relevant is WGS84 for topographic maps? Global Standardization: Compatibility with GPS: Interoperability: Online Mapping Services: Geographic Information Systems (GIS): International Collaboration: Precision in Elevation Data: Satellite Technology Integration: Standardization in Topographic Map Production: Adaptability to Local Coordinate Systems: What coordinate systems are using WGS84? Geographic Coordinates (Latitude and Longitude): Geodetic Coordinates (Latitude, Longitude, and Ellipsoidal Height): Cartesian Coordinates (X, Y, and Z): UTM (Universal Transverse Mercator) Coordinates: MGRS (Military Grid Reference System) Coordinates: ECEF (Earth-Centered, Earth-Fixed) Coordinates: GPS Coordinates: What is an ellipsoid, and why is it essential for coordinate systems? Key Characteristics of an Ellipsoid: Semi-Major and Semi-Minor Axes: Flattening (F): Eccentricity (e): Importance of Ellipsoids in Coordinate Systems: Earth\u0026rsquo;s Geoid Deviation: Accuracy in Distance Measurements: Precision in Geographic Coordinates: Consistency in Global Positioning: Uniformity in Coordinate Systems: Altitude and Elevation Accuracy: Conclusion: Introduction: Understanding Geodetic Reference Systems # The Earth is not a perfect sphere; it is an oblate spheroid, meaning its shape is slightly flattened at the poles and bulging at the equator. Geodetic reference systems accurately represent the Earth\u0026rsquo;s surface for various purposes, such as navigation, mapping, and scientific research.\nA geodetic reference system is a framework that provides a consistent and standardized way to measure and represent locations on Earth\u0026rsquo;s surface. These systems involve a set of parameters, models, and coordinate systems that define the size and shape of the Earth, as well as the reference point from which measurements are made.\nHistorical Evolution of Geodetic Reference Systems # The need for accurate geodetic reference systems has been recognized for centuries. Early attempts at creating such systems involved simple models and assumptions about the Earth\u0026rsquo;s shape. The development of more sophisticated reference systems gained momentum with advancements in geodesy, the science of measuring the Earth.\nOne crucial milestone in this evolution was the establishment of the World Geodetic System in 1984, commonly known as WGS 84. It significantly improved over its predecessors, incorporating advanced technologies and a more comprehensive understanding of the Earth\u0026rsquo;s shape.\nWGS 84 Basic Technical Specifications # WGS 84 is based on a three-dimensional Cartesian coordinate system. Its specifications include the semi-major axis, flattening, and gravitational constants. The semi-major axis represents the Earth\u0026rsquo;s equatorial radius, while flattening characterizes the deviation from a perfect sphere.\nThe gravitational constants in WGS 84 describe the mass distribution within the Earth, affecting the gravitational field. These constants are crucial in determining the geoid, an equipotential surface approximating mean sea level.\nWGS 84 coordinates are expressed in latitude, longitude, and ellipsoidal height. Latitude measures the north-south position, longitude the east-west position, and ellipsoidal height the distance above the reference ellipsoid. This coordinate system is essential for accurate positioning and navigation on Earth\u0026rsquo;s surface.\nSatellite Contributions to WGS 84 # Satellite technology has played a pivotal role in developing and maintaining WGS 84. The Global Positioning System (GPS), a constellation of satellites orbiting the Earth, provides precise and continuous signals that enable accurate positioning. WGS 84 coordinates are widely used in GPS devices for navigation, surveying, and various scientific applications.\nIntegrating satellite measurements into WGS 84 involves complex calculations and adjustments to ensure consistency with ground-based observations. This collaborative effort between ground-based geodetic measurements and satellite technology enhances the accuracy and reliability of WGS 84.\nWGS 84 and Global Positioning System (GPS) # WGS 84 and GPS are closely intertwined, with WGS 84 as the reference system for GPS coordinates. The GPS receivers in devices like smartphones, navigation systems, and surveying equipment use signals from multiple satellites to determine the user\u0026rsquo;s precise location regarding WGS 84 coordinates.\nThe widespread adoption of GPS technology has transformed navigation, geolocation services, and various industries reliant on accurate positioning information. Emergency services, transportation, agriculture, and scientific research benefit from the seamless integration of WGS 84 and GPS.\nPractical Applications of WGS 84 # The applications of WGS 84 extend far beyond navigation and GPS. Mapping and cartography heavily rely on this reference system to represent geographical features accurately. Geographic Information Systems (GIS) use WGS 84 coordinates to organize and analyze spatial data, facilitating informed decision-making in diverse fields.\nWGS 84 is the standard for aeronautical navigation and flight planning in aviation. The precise coordinates of this reference system provide enhanced safety and efficiency in air travel. Similarly, maritime navigation relies on WGS 84 to ensure accurate positioning at sea.\nGeoscientific research, including studies on Earth\u0026rsquo;s gravity field, sea level changes, and tectonic plate movements, benefits from the consistency and accuracy of WGS 84. The reference system provides a common framework for comparing and analyzing data from various sources worldwide.\nChallenges and Evolving Standards # While WGS 84 has been a groundbreaking reference system, it has challenges. The Earth\u0026rsquo;s dynamic nature, with its shape and gravitational field, changes over time and requires periodic updates to the reference system. The evolution of technology and improvements in measurement techniques also necessitate adjustments to maintain the accuracy of WGS 84.\nInternational collaboration is essential for the continuous refinement of geodetic reference systems. Organizations such as the International Association of Geodesy (IAG) and the International Earth Rotation and Reference Systems Service (IERS) are critical in coordinating efforts to enhance the precision and consistency of global geodetic reference frames.\nConclusion: WGS 84 in a Global Context # In conclusion, WGS 84 is a cornerstone in geodetic reference systems, providing a standardized framework for accurate location representation on Earth. Its development marked a significant advancement in geodesy, combining ground-based measurements with satellite technology to create a robust and globally accepted coordinate system.\nThe integration of WGS 84 with GPS has revolutionized navigation and positioning, impacting diverse sectors ranging from transportation to scientific research. As technology advances and our understanding of Earth\u0026rsquo;s dynamic nature deepens, WGS 84 will continue to evolve, ensuring that it remains a reliable and indispensable global spatial reference tool.\nHow to use WGS84? # Using the WGS 84 coordinate system involves understanding its parameters, obtaining coordinates, and applying them in various contexts. Here\u0026rsquo;s a step-by-step guide on how to use WGS 84:\nUnderstand WGS 84 Parameters: # Familiarize yourself with the critical parameters of WGS 84, including the semi-major axis, flattening, and gravitational constants. These values define the shape and size of the Earth in the WGS 84 reference system.\nCoordinate Representation: # WGS 84 coordinates are typically represented in latitude, longitude, and ellipsoidal height. Latitude measures the north-south position, longitude measures the east-west position, and ellipsoidal height represents the distance above the reference ellipsoid.\nUse GPS Devices: # Many consumer devices, such as smartphones and GPS navigation systems, use WGS 84 coordinates for location services. Ensure that your device is set to use WGS 84 as the coordinate reference system.\nMapping and GIS Applications: # In mapping and Geographic Information Systems (GIS), WGS 84 coordinates are commonly used to represent spatial data. GIS software allows you to input or analyze data using WGS 84 coordinates, enabling accurate mapping and geospatial analysis.\nAviation and Maritime Navigation: # If you\u0026rsquo;re involved in aviation or maritime activities, WGS 84 is the standard for navigation. GPS devices on aircraft and ships use WGS 84 coordinates for accurate positioning and route planning.\nSurveying and Geodetic Measurements: # Surveyors and geodesists use WGS 84 coordinates for precise measurements of land, boundaries, and geodetic control points. Ensure that your surveying equipment is configured to use WGS 84.\nConversion Tools: # If you need to convert coordinates between different reference systems, use conversion tools or software. WGS 84 coordinates can be converted to other systems and vice versa to ensure compatibility with various applications.\nOnline Mapping Services: # Many online mapping services, such as Google Maps, use WGS 84 coordinates by default. When interacting with these services, understand that the displayed coordinates are likely in the WGS 84 system.\nProgramming and APIs: # If you\u0026rsquo;re a developer, understand how to work with WGS 84 coordinates in your programming language of choice. Many programming libraries and APIs provide functions for working with spatial data using WGS 84.\nStay Informed about Updates: # WGS 84 is periodically updated to account for changes in Earth\u0026rsquo;s shape and gravitational field. Stay informed about updates and ensure that your systems and devices are using the latest parameters for WGS 84.\nCollaboration with Others: # When sharing or receiving spatial data, ensure that all parties use the same coordinate reference system, preferably WGS 84. This ensures consistency and accuracy in data interpretation and analysis.\nFollowing these steps, you can effectively use the WGS 84 coordinate system in various applications, from everyday navigation to professional surveying and geospatial analysis.\nHow relevant is WGS84 for topographic maps? # The World Geodetic System 1984 (WGS 84) is highly relevant for topographic maps due to its widespread adoption as the gold global positioning and mapping standard. Here\u0026rsquo;s why WGS 84 is crucial in the context of topographic maps:\nGlobal Standardization: # WGS 84 serves as a global standard for geodetic reference, providing a consistent and uniform framework for global mapping. This standardization is crucial for international collaboration, data sharing, and seamless integration of maps from different sources.\nCompatibility with GPS: # Topographic maps often involve fieldwork, surveying, and navigation. WGS 84 coordinates are directly compatible with GPS systems, which have become integral tools for collecting and verifying topographic data. GPS receivers provide real-time positioning information based on WGS 84 coordinates.\nInteroperability: # WGS 84 facilitates interoperability between various mapping systems and technologies. As a widely accepted standard, it ensures that topographic data collected in one location can be easily integrated and compared with data from other regions, even if different mapping tools or devices were used.\nOnline Mapping Services: # Many online mapping services, including Google Maps and OpenStreetMap, use WGS 84 coordinates. Topographic maps accessed through these platforms are typically based on WGS 84, providing a user-friendly and consistent experience for individuals and professionals alike.\nGeographic Information Systems (GIS): # GIS, commonly used in topography and cartography, often employs WGS 84 as the default coordinate system. This ensures that spatial data, including topographic information, is accurately represented and analyzed within GIS software.\nInternational Collaboration: # Topographic maps frequently involve data from multiple countries and regions. WGS 84\u0026rsquo;s global standardization facilitates international collaboration in creating and sharing topographic information, ensuring that maps align seamlessly across borders.\nPrecision in Elevation Data: # Topographic maps include elevation information, and WGS 84 provides a precise framework for representing elevations. The ellipsoidal height component in WGS 84 coordinates accounts for variations in the Earth\u0026rsquo;s shape, enhancing the accuracy of elevation data on topographic maps.\nSatellite Technology Integration: # WGS 84 is integrated with satellite technologies like the Global Positioning System (GPS). This integration enhances the accuracy of location-based data on topographic maps, making them valuable tools for navigation, exploration, and scientific research.\nStandardization in Topographic Map Production: # Many national mapping agencies and organizations producing topographic maps use WGS 84 as the reference system. This standardization streamlines the map production process and ensures consistency in data representation across different regions.\nAdaptability to Local Coordinate Systems: # While WGS 84 is a global standard, topographic maps may also use local or regional coordinate systems for specific projects. Conversion tools allow for transforming WGS 84 coordinates to local systems when needed, maintaining accuracy in local mapping applications.\nIn summary, WGS 84\u0026rsquo;s relevance in topographic maps is multifaceted, encompassing global standardization, compatibility with GPS and online mapping services, GIS support, and satellite technology integration. Its adoption ensures that topographic maps provide accurate and consistent spatial information, facilitating various applications in surveying, navigation, environmental monitoring, and more.\nWhat coordinate systems are using WGS84? # The World Geodetic System 1984 (WGS 84) is commonly used with various coordinate systems for representing positions on the Earth\u0026rsquo;s surface. The primary coordinate systems associated with WGS 84 include:\nGeographic Coordinates (Latitude and Longitude): # The most fundamental coordinate system associated with WGS 84 is geographic coordinates, representing positions on the Earth\u0026rsquo;s surface in terms of latitude and longitude. Latitude measures north-south position, while longitude measures east-west position. This coordinate system is widely used in mapping, navigation, and geospatial applications.\nGeodetic Coordinates (Latitude, Longitude, and Ellipsoidal Height): # In addition to latitude and longitude, WGS 84 includes a vertical component called ellipsoidal height. Geodetic coordinates, expressed as latitude, longitude, and ellipsoidal height, represent a point on the Earth\u0026rsquo;s surface. This is particularly important for applications involving elevation data.\nCartesian Coordinates (X, Y, and Z): # WGS 84 can also be expressed in Cartesian coordinates, where the position of a point is represented by its X, Y, and Z coordinates in a three-dimensional Cartesian coordinate system. The X and Y coordinates correspond to the east-west and north-south directions, while the Z coordinate represents the height above the reference ellipsoid.\nUTM (Universal Transverse Mercator) Coordinates: # The Universal Transverse Mercator coordinate system divides the world into a series of zones, each with its own coordinate system. WGS 84 is commonly used as the reference ellipsoid for UTM coordinates. UTM coordinates include easting, northing, zone number, and hemisphere information, providing a localized Cartesian coordinate system for more accurate measurements over smaller areas.\nMGRS (Military Grid Reference System) Coordinates: # MGRS is a standardized system for expressing locations on the Earth\u0026rsquo;s surface. It is based on the UTM coordinate system and uses a combination of letters and numbers to represent grid squares. WGS 84 is often used as the underlying reference for MGRS coordinates.\nECEF (Earth-Centered, Earth-Fixed) Coordinates: # ECEF coordinates are a three-dimensional Cartesian coordinate system centred at the Earth\u0026rsquo;s centre. WGS 84 is the reference ellipsoid for ECEF coordinates, allowing precise representation of points in a global, Earth-centered framework.\nGPS Coordinates: # The Global Positioning System (GPS) uses WGS 84 as its reference system. GPS coordinates, expressed in terms of latitude, longitude, and sometimes altitude, are based on the WGS 84 ellipsoid. This ensures compatibility and consistency when using GPS devices for navigation and location-based services.\nThese coordinate systems, all based on the WGS 84 reference ellipsoid, are widely used in various applications, including mapping, navigation, surveying, geospatial analysis, and scientific research. Adopting WGS 84 as a global standard promotes interoperability and consistency in spatial data representation across different systems and devices.\nWhat is an ellipsoid, and why is it essential for coordinate systems? # An ellipsoid, also known as a spheroid, is a three-dimensional geometric figure that closely approximates the shape of the Earth. It is formed by rotating an ellipse (a flattened circle) about its shorter or longer axis. The resulting shape is similar to a sphere but is slightly flattened at the poles and bulging at the equator. This deviation from a perfect sphere is due to the Earth\u0026rsquo;s rotation and the gravitational forces acting on it.\nKey Characteristics of an Ellipsoid: # Semi-Major and Semi-Minor Axes: # An ellipsoid is characterized by the semi-major axis (a) and the semi-minor axis (b). The semi-major axis corresponds to the equatorial radius, while the semi-minor axis corresponds to the polar radius.\nFlattening (F): # Flattening measures how much the ellipsoid deviates from a perfect sphere. It is calculated as the difference between the semi-major and semi-minor axes divided by the semi-major axis (F = (a - b) / a). The flattening value determines the degree of ellipsoidal flattening.\nEccentricity (e): # Eccentricity is another parameter that describes the shape of an ellipsoid. It is related to flattening and is calculated as the square root of (1 - (b²/a²)). Eccentricity measures how much the ellipse deviates from a perfect circle.\nImportance of Ellipsoids in Coordinate Systems: # Earth\u0026rsquo;s Geoid Deviation: # The Earth\u0026rsquo;s actual shape is more accurately represented by an ellipsoid than a perfect sphere. An ellipsoidal model better approximates the Earth\u0026rsquo;s accurate dimensions, accounting for the flattening at the poles and bulging at the equator.\nAccuracy in Distance Measurements: # Ellipsoidal models are crucial for accurate distance measurements on the Earth\u0026rsquo;s surface. Using a sphere for coordinate systems would introduce errors, especially over large distances, as it would not account for the variations in radius associated with an ellipsoidal shape.\nPrecision in Geographic Coordinates: # Geographic coordinates, such as latitude and longitude, are specified with an ellipsoid. Ellipsoidal models, like the one defined by WGS 84, ensure that positions on the Earth\u0026rsquo;s surface are exact, providing a standard reference for mapping, navigation, and geospatial applications.\nConsistency in Global Positioning: # Satellite-based navigation systems like GPS use ellipsoidal models like WGS 84. This ensures that positions obtained from GPS receivers are compatible with mapping systems and coordinate reference frames globally.\nUniformity in Coordinate Systems: # Adopting ellipsoidal models, particularly WGS 84, promotes worldwide standardization and uniformity in coordinate systems. This consistency facilitates data exchange and interoperability across different mapping and geospatial applications.\nAltitude and Elevation Accuracy: # For applications involving altitude or elevation measurements, an ellipsoidal model is essential. The ellipsoidal height component in coordinates represents the distance above or below the reference ellipsoid, providing accurate elevation information.\nIn summary, ellipsoids are crucial in coordinate systems because they accurately represent the Earth\u0026rsquo;s shape more accurately than a simple sphere. Using ellipsoidal models, coordinate systems can better reflect the complex geoid structure, ensuring precision in location-based data and maintaining consistency in global positioning applications.\nConclusion: # The World Geodetic System 1984 (WGS 84) is a geodetic reference system widely used in various coordinate systems, including geographic coordinates (latitude and longitude), geodetic coordinates (latitude, longitude, and ellipsoidal height), Cartesian coordinates (X, Y, and Z), Universal Transverse Mercator (UTM) coordinates, Military Grid Reference System (MGRS) coordinates, Earth-Centered, Earth-Fixed (ECEF) coordinates, and GPS coordinates. WGS 84, based on an ellipsoidal model, is crucial for global standardization, GPS compatibility, mapping system interoperability, and elevation data precision. It plays a significant role in topographic maps, mapping and GIS applications, aviation, maritime navigation, surveying, and scientific research. The adoption of WGS 84 ensures consistency in spatial data representation and supports accurate positioning on a global scale. The ellipsoidal shape of the Earth is vital for precise distance measurements, accuracy in geographic coordinates, and maintaining uniformity in coordinate systems, making it a fundamental component of geospatial sciences and navigation technologies.\nHappy Navigating ;-)\nSven\n","date":"18 December 2023","externalUrl":null,"permalink":"/posts/what-is-wgs84-an-overview/","section":"Posts","summary":"The World Geodetic System 1984 (WGS 84) is a geodetic reference system used to describe the shape and size of the Earth. We will delve into its historical context, technical specifications, global significance, and practical applications.\n","title":"What is WGS84 - An Overview","type":"posts"},{"content":" What is - Input Validation? # Input validation is a process used to ensure that the data provided to a system or application meets specific criteria or constraints before it is accepted and processed. The primary goal of input validation is to improve the reliability and security of a system by preventing invalid or malicious data from causing errors or compromising the system\u0026rsquo;s integrity.\nWhat is - Input Validation? What are the critical aspects of input validation: Data Integrity: Security: User Experience: Compliance: Preventing Bugs: What does input validation have to do with security? Preventing Injection Attacks: Avoiding Buffer Overflows: Mitigating Cross-Site Request Forgery (CSRF) Attacks: Protecting Against Data Tampering: Enhancing Authentication and Authorisation: Securing File Operations: Preventing Denial-of-Service (DoS) Attacks: Ensuring Data Integrity: Where can input validation take place in a program? User Interface (UI) Level: Server-Side Level: Database Level: API Level: Middleware Level: Configuration Files: Logging Level: Web Application Firewall (WAF): What are the advantages and disadvantages of server-side and client-side input validation? Server-Side Input Validation Advantages: Security: Data Integrity: Business Logic: User Experience: Cross-Browser Compatibility: Code Reusability: Disadvantages: Response Time: User Experience: Bandwidth Usage: Server Load: Client-Side Input Validation Advantages: Immediate Feedback: Bandwidth Efficiency: Enhanced User Experience: Reduced Server Load: Reduced Round-Trip Time: Disadvantages: Security Risks: Browser Dependency: Code Duplication: Accessibility: Browser Performance: Lessons Learned - client- versus server-side validation: What are standard techniques for input validation? Data Type Checks: Length Checks: Format Checks: Range Checks: Pattern Matching: Allowlisting/Blocklisting: Conclusion: What are the critical aspects of input validation: # Data Integrity: # Input validation helps maintain data integrity by ensuring it conforms to the expected format and type. For example, if a system expects a numerical input, input validation can check whether the entered data is a number.\nSecurity: # Proper input validation is crucial for preventing security vulnerabilities such as injection attacks (e.g., SQL injection, cross-site scripting) where malicious data is inserted to exploit vulnerabilities in a system.\nUser Experience: # Input validation can also contribute to a better user experience by providing immediate feedback to users about the correctness of their input. Instead of allowing users to submit invalid data and encounter errors, input validation helps users correct mistakes upfront.\nCompliance: # In some industries, regulatory requirements mandate proper input validation to protect sensitive information and ensure the appropriate functioning of systems.\nPreventing Bugs: # Input validation can help prevent bugs and unexpected behaviour in software by ensuring that the data flowing through the system adheres to the expected standards.\nWhat does input validation have to do with security? # Input validation is crucial for security in software development because it helps prevent a wide range of security vulnerabilities and attacks. Here are some key reasons why input validation is essential for security:\nPreventing Injection Attacks: # Input validation helps protect against injection attacks, where malicious code is injected into user input. Common examples include SQL injection and cross-site scripting (XSS) attacks. By validating and sanitising input, developers can ensure that user input doesn\u0026rsquo;t contain malicious code the application could execute.\nAvoiding Buffer Overflows: # Input validation helps prevent buffer overflows, a vulnerability where an attacker can exploit the program\u0026rsquo;s memory by providing input that exceeds the allocated buffer size. By validating input size and content, developers can mitigate the risk of buffer overflow attacks.\nMitigating Cross-Site Request Forgery (CSRF) Attacks: # CSRF attacks involve tricking a user\u0026rsquo;s browser into making unintended requests to a web application on which the user is authenticated. Proper input validation ensures that requests originate from the expected sources, reducing the risk of CSRF attacks.\nProtecting Against Data Tampering: # Input validation helps protect against data tampering by ensuring that the data received by the application adheres to expected formats and constraints. This prevents attackers from manipulating data to exploit vulnerabilities or gain unauthorised access.\nEnhancing Authentication and Authorisation: # Validating input is essential for enforcing proper authentication and authorisation checks. Incorrect or manipulated input could bypass authentication mechanisms or grant unauthorised access to sensitive areas of an application.\nSecuring File Operations: # Input validation is crucial when dealing with file uploads or file-related operations. Without proper validation, an attacker might upload malicious files or manipulate file names to execute arbitrary code on the server.\nPreventing Denial-of-Service (DoS) Attacks: # Input validation can mitigate the impact of DoS attacks by rejecting or limiting input that could lead to resource exhaustion. For example, limiting the length of input or the rate of requests can help protect against certain types of DoS attacks.\nEnsuring Data Integrity: # Input validation contributes to data integrity by ensuring that the data processed by an application meets expected standards. This is important for preventing unintentional errors and maintaining the consistency and reliability of the system.\nIn summary, input validation is a fundamental security practice that helps build robust and resilient software systems. By validating and sanitising input data, developers can significantly reduce the risk of common security vulnerabilities and enhance the overall security posture of their applications.\nWhere can input validation take place in a program? # Input validation is a crucial aspect of writing secure and robust software. It helps ensure that the data your program receives is of the expected format and within acceptable ranges. Input validation can take place at various levels in a program, including:\nUser Interface (UI) Level: # Client-Side Validation: It can be performed using JavaScript or other client-side scripting languages in web applications to provide immediate feedback to users. However, client-side validation should not be solely relied upon for security, as malicious users can bypass it.\nServer-Side Level: # Server-Side Validation: This is the most critical layer for input validation. All input received from the client should be validated on the server to ensure that it conforms to the expected format, length, and other criteria. Server-side validation is more secure than client-side validation because it cannot be easily bypassed or tampered with by users.\nDatabase Level: # Database Input Validation: Before storing data in a database, validating the input is essential to prevent SQL injection attacks and ensure data integrity. This can involve using parameterised queries or prepared statements to prevent malicious input from affecting the database.\nAPI Level: # API Input Validation: If your application interacts with external APIs, it\u0026rsquo;s crucial to validate input data before sending requests. This helps prevent injection attacks and ensures the API receives the expected data.\nMiddleware Level: # Middleware Validation: Middleware components may process or filter incoming data in some applications before it reach the core application logic. Input validation can be performed at this level to ensure only valid data is passed to the application.\nConfiguration Files: # Configuration Input Validation: If your program reads configuration files, validate the input data to ensure it adheres to the expected format and doesn\u0026rsquo;t introduce vulnerabilities.\nLogging Level: # Logging Input Validation: When logging user inputs or other sensitive information, ensure that the logged data is properly validated to prevent accidental exposure of sensitive information or injection attacks through log manipulation.\nWeb Application Firewall (WAF): # WAF Validation: WAFs can filter and validate incoming HTTP traffic. They can help protect against common web application vulnerabilities, including input-related attacks.\nBy implementing input validation at multiple levels, you create a defence-in-depth strategy that enhances the security of your application. Always validate input data based on your application\u0026rsquo;s requirements and constraints to ensure proper functionality and security.\nWhat are the advantages and disadvantages of server-side and client-side input validation? # At this point, I\u0026rsquo;ll briefly overview the advantages and disadvantages of server-side and client-side input validation, covering various aspects of security, performance, user experience, and maintenance.\nServer-Side Input Validation # Advantages: # Security: # Protection Against Malicious Input : Server-side validation is crucial for security as it prevents malicious input from reaching the database or application logic. This is especially important for avoiding SQL injection, cross-site scripting (XSS), and other security vulnerabilities.\nCentralised Control : Centralised validation on the server allows for consistent and thorough input data validation, reducing the risk of overlooking potential security threats.\nData Integrity: # Consistent Data Quality : Server-side validation ensures that the data entering the system meets the specified criteria, maintaining data integrity and consistency throughout the application.\nPrevention of Data Corruption : The risk of corrupted or invalid data entering the database is minimised by validating input on the server.\nBusiness Logic: # Enforcement of Business Rules : Server-side validation is essential for enforcing business rules and logic. This ensures that data conforms not only to technical specifications but also to the application\u0026rsquo;s specific requirements.\nUser Experience: # Consistent Validation Messages : Server-side validation provides the opportunity to generate standardised error messages, creating a more consistent and user-friendly experience for end-users.\nCross-Browser Compatibility: # Browser Independence : Server-side validation is not dependent on the user\u0026rsquo;s browser, making it more reliable and consistent across different browser environments.\nCode Reusability: # Centralised Validation Logic : Server-side validation allows for the centralisation of validation logic, promoting code reusability across multiple application parts.\nDisadvantages: # Response Time: # Increased Round-Trip Time : Server-side validation requires a round-trip to the server, leading to potential delays in response time, especially in applications with high latency.\nUser Experience: # Delayed Feedback : Users may experience delays in receiving feedback on their input, leading to a less responsive and potentially frustrating user experience.\nBandwidth Usage: # Increased Bandwidth Usage : Each validation request sent to the server consumes bandwidth, which can be a concern in bandwidth-sensitive environments or for users with limited data plans.\nServer Load: # Increased Server Load : Server-side validation contributes to server load, and in high-traffic applications, this can lead to resource contention and decreased overall performance.\nClient-Side Input Validation # Advantages: # Immediate Feedback: # Real-time Validation : Client-side validation provides immediate feedback to users, improving the application\u0026rsquo;s overall responsiveness and perceived speed.\nBandwidth Efficiency: # Reduced Server Requests : Client-side validation minimises the number of requests to the server, conserving bandwidth and improving overall network efficiency.\nEnhanced User Experience: # Responsive Interactivity : By validating input on the client side, users receive instant feedback, creating a more interactive and responsive user experience.\nReduced Server Load: # Offloading Server Processing : Client-side validation offloads some processing tasks from the server, reducing the overall load and improving server performance.\nReduced Round-Trip Time: # Faster Form Submissions : Since validation occurs on the client side, users experience speedier form submissions without waiting for a round-trip to the server.\nDisadvantages: # Security Risks: # Vulnerability to Manipulation : Client-side validation is susceptible to manipulation by malicious users who can bypass or disable the client-side validation to submit malicious input directly to the server.\nBrowser Dependency: # Inconsistent Browser Support : Client-side validation may need consistent support across different browsers, leading to behaviour and user experience variations.\nCode Duplication: # Duplication of Validation Logic : Client-side validation requires duplicating validation logic on the server to ensure data integrity and security, leading to potential maintenance challenges.\nAccessibility: # Accessibility Concerns : Relying solely on client-side validation may pose accessibility challenges for disabled users using alternative input methods or assistive technologies.\nBrowser Performance: # Resource Intensive : Intensive client-side validation logic can consume significant browser resources, impacting the performance of the user\u0026rsquo;s device.\nLessons Learned - client- versus server-side validation: # In conclusion, the choice between server-side and client-side input validation depends on various factors such as security requirements, user experience goals, and application architecture. A balanced approach often involves a combination of both server-side and client-side validation to harness the advantages of each while mitigating their respective disadvantages. Careful consideration of the specific needs and constraints of the application is crucial to implementing a practical and secure input validation strategy.\nWhat are standard techniques for input validation? # Data Type Checks: # Ensuring that the entered data matches the expected data type (e.g., numbers, dates, strings).\nHere\u0026rsquo;s a simple Java example using the instanceof operator, which checks whether an object is an instance of a particular class or interface:\npublic class DataTypeCheckExample { public static void main(String[] args) { // Example with an Integer Object value = 42; if (value instanceof Integer) { int intValue = (Integer) value; System.out.println(\u0026#34;The value is an Integer: \u0026#34; + intValue); } else { System.out.println(\u0026#34;The value is not an Integer.\u0026#34;); } // Example with a String Object anotherValue = \u0026#34;Hello, Java!\u0026#34;; if (anotherValue instanceof String) { String stringValue = (String) anotherValue; System.out.println(\u0026#34;The value is a String: \u0026#34; + stringValue); } else { System.out.println(\u0026#34;The value is not a String.\u0026#34;); } } } In this example, we have an Object variable (value and anotherValue) that can hold any object. We use the instanceof operator to check whether the object is an instance of a specific type (Integer or String). We can safely cast it to that type and perform operations accordingly if it is. If not, we handle it appropriately.\nRemember, it\u0026rsquo;s generally a good practice to use proper data types whenever possible rather than relying on type checks, but there are situations where type checks are necessary.\nLength Checks: # Verifying that the length of the input is within acceptable limits.\nIn Java, you can perform length checks on various data structures, such as strings or arrays, to ensure they meet specific criteria. Here\u0026rsquo;s an example of length checks for a string in Java:\npublic class LengthCheckExample { public static void main(String[] args) { //example string String myString = \u0026#34;Hello, World!\u0026#34;; // Check the length of the string int length = myString.length(); // Perform length checks if (length \u0026gt; 0) { System.out.println(\u0026#34;The string is not empty.\u0026#34;); } else { System.out.println(\u0026#34;The string is empty.\u0026#34;); } if (length \u0026gt;= 10) { System.out.println(\u0026#34;The string is at least 10 characters long.\u0026#34;); } else { System.out.println(\u0026#34;The string is less than 10 characters long.\u0026#34;); } } } In this example, we use the length() method of the String class to get the length of the string. We then perform two length checks: one to determine if the string is empty and another to check if it\u0026rsquo;s at least ten characters long.\nRemember that the specific length of checks you perform will depend on the requirements of your program or application. You can adapt similar checks for arrays or other data structures as needed.\nFormat Checks: # Ensuring the input follows a specified format (e.g., email addresses, phone numbers).\nBelow is an example Java program that performs format checks for email addresses and phone numbers using regular expressions:\nimport java.util.regex.Matcher; import java.util.regex.Pattern; public class FormatChecker { public static void main(String[] args) { // Example usage String email = \u0026#34;user@example.com\u0026#34;; String phoneNumber = \u0026#34;+1-555-555-5555\u0026#34;; if (isValidEmail(email)) { System.out.println(\u0026#34;Email is valid: \u0026#34; + email); } else { System.out.println(\u0026#34;Email is not valid: \u0026#34; + email); } if (isValidPhoneNumber(phoneNumber)) { System.out.println(\u0026#34;Phone number is valid: \u0026#34; + phoneNumber); } else { System.out.println(\u0026#34;Phone number is not valid: \u0026#34; + phoneNumber); } } // Check if the email address is valid public static boolean isValidEmail(String email) { String emailRegex = \u0026#34;^[a-zA-Z0-9_+\u0026amp;*-]+(?:\\\\.[a-zA-Z0-9_+\u0026amp;*-]+)*@(?:[a-zA-Z0-9-]+\\\\.)+[a-zA-Z]{2,7}$\u0026#34;; Pattern pattern = Pattern.compile(emailRegex); Matcher matcher = pattern.matcher(email); return matcher.matches(); } // Check if the phone number is valid public static boolean isValidPhoneNumber(String phoneNumber) { // Accepts numbers in the format: +1-555-555-5555 String phoneRegex = \u0026#34;^\\\\+(?:[0-9] ?){6,14}[0-9]$\u0026#34;; Pattern pattern = Pattern.compile(phoneRegex); Matcher matcher = pattern.matcher(phoneNumber); return matcher.matches(); } } In this example, isValidEmail and isValidPhoneNumber are two methods that use regular expressions to check if the provided email address and phone number are in the expected format. The regular expressions in this example may need adjustment based on your specific format requirements. Regular expressions are powerful tools for pattern matching but can be complex. It\u0026rsquo;s essential to thoroughly test them with various inputs to ensure they meet your needs.\nIf you prefer not to use regular expressions, you can perform format checks for email addresses and phone numbers using other string manipulation techniques. Here\u0026rsquo;s an example:\npublic class FormatChecker { public static void main(String[] args) { // Example usage String email = \u0026#34;user@example.com\u0026#34;; String phoneNumber = \u0026#34;+1-555-555-5555\u0026#34;; if (isValidEmail(email)) { System.out.println(\u0026#34;Email is valid: \u0026#34; + email); } else { System.out.println(\u0026#34;Email is not valid: \u0026#34; + email); } if (isValidPhoneNumber(phoneNumber)) { System.out.println(\u0026#34;Phone number is valid: \u0026#34; + phoneNumber); } else { System.out.println(\u0026#34;Phone number is not valid: \u0026#34; + phoneNumber); } } // Check if the email address is valid public static boolean isValidEmail(String email) { int atIndex = email.indexOf(\u0026#39;@\u0026#39;); int dotIndex = email.lastIndexOf(\u0026#39;.\u0026#39;); // Ensure there is exactly one @ symbol, and it is not the first or last character if (atIndex \u0026gt; 0 \u0026amp;\u0026amp; atIndex \u0026lt; email.length() - 1) { // Ensure there is at least one dot after the @ symbol if (dotIndex \u0026gt; atIndex \u0026amp;\u0026amp; dotIndex \u0026lt; email.length() - 1) { return true; } } return false; } // Check if the phone number is valid public static boolean isValidPhoneNumber(String phoneNumber) { // Check the length and specific characters if (phoneNumber.length() \u0026gt;= 10 \u0026amp;\u0026amp; phoneNumber.charAt(0) == \u0026#39;+\u0026#39; \u0026amp;\u0026amp; phoneNumber.charAt(1) \u0026gt;= \u0026#39;1\u0026#39; \u0026amp;\u0026amp; phoneNumber.charAt(1) \u0026lt;= \u0026#39;9\u0026#39;) { // Check that the rest of the characters are digits or hyphens for (int i = 2; i \u0026lt; phoneNumber.length(); i++) { char c = phoneNumber.charAt(i); if (!(Character.isDigit(c) || c == \u0026#39;-\u0026#39;)) { return false; } } return true; } return false; } } This example uses basic string manipulation and checks for specific conditions rather than regular expressions. Remember that regular expressions are often a more concise and powerful way to perform such checks, but this alternative approach may be suitable for more straightforward cases.\nRange Checks: # Verifying that numerical input falls within a predefined range.\nYou can perform range checks using conditional statements. Here\u0026rsquo;s a simple example that demonstrates range checks for a given value within a specified range:\npublic class RangeCheckExample { public static void main(String[] args) { // Define the range int lowerBound = 10; int upperBound = 20; // Test values int value1 = 15; int value2 = 5; int value3 = 25; // Perform range checks System.out.println(\u0026#34;Value \u0026#34; + value1 + \u0026#34; is in range: \u0026#34; + isInRange(value1, lowerBound, upperBound)); System.out.println(\u0026#34;Value \u0026#34; + value2 + \u0026#34; is in range: \u0026#34; + isInRange(value2, lowerBound, upperBound)); System.out.println(\u0026#34;Value \u0026#34; + value3 + \u0026#34; is in range: \u0026#34; + isInRange(value3, lowerBound, upperBound)); } // Function to check if a value is within a specified range private static boolean isInRange(int value, int lowerBound, int upperBound) { return value \u0026gt;= lowerBound \u0026amp;\u0026amp; value \u0026lt;= upperBound; } } In this example, the isInRange function takes a value and two bounds (lower and upper). It returns true if the value is within the specified range (inclusive) and false otherwise. The main method demonstrates the usage of this function for three different values and prints whether each value is in the specified range.\nPattern Matching: # Checking input against a predefined pattern or regular expression.\nPattern matching was introduced in Java 16 as a preview feature and became standard in Java 17. Here\u0026rsquo;s a simple example using pattern matching with the old instanceof operator:\npublic class PatternMatchingExample { public static void main(String[] args) { Object value = \u0026#34;Hello, Pattern Matching!\u0026#34;; if (value instanceof String str) { System.out.println(\u0026#34;Length of the string: \u0026#34; + str.length()); } else { System.out.println(\u0026#34;Not a string\u0026#34;); } } } In this example, we have an Object called value, and we want to check if it is an instance of String. If it is, we introduce a new variable str of type String in the same if statement and can use it within that block. This eliminates the need for an additional cast and makes the code more concise.\nIn Java 16, pattern matching was introduced as a preview feature, and while instanceof was commonly used, there were no specific pattern-matching constructs. However, one common pattern-matching use case was in switch expressions. Here\u0026rsquo;s an example:\npublic class PatternMatchingExample { public static void main(String[] args) { Object value = \u0026#34;Hello, Pattern Matching!\u0026#34;; int result = switch (value) { case String str -\u0026gt; str.length(); case Integer num -\u0026gt; num * 2; default -\u0026gt; 0; }; System.out.println(\u0026#34;Result: \u0026#34; + result); } } In this example, the switch expression performs pattern matching. Different patterns are matched depending on the type of the value. If it\u0026rsquo;s a String, the length of the string is returned. If it\u0026rsquo;s an Integer, the value is doubled. The default case is used if the value doesn\u0026rsquo;t match any of the specified patterns.\nNote that pattern matching with instanceof is more explicitly available from Java 17 onwards. But this will be a separate article.\nAllowlisting/Blocklisting: # Allowing or disallowing specific characters or patterns in the input.\nAllowlisting and blocklisting are often used in the context of security to control access to specific resources or actions. In Java, you can implement allowlisting or blocklisting using various approaches. Below is a simple example demonstrating how you might implement a basic allowlist and blocklist for a hypothetical resource.\nLet\u0026rsquo;s say we have a class representing a resource and want to control access based on a list of allowed and disallowed entities.\nimport java.util.ArrayList; import java.util.List; class ResourceAccessController { private List\u0026lt;String\u0026gt; whitelist; private List\u0026lt;String\u0026gt; blacklist; public ResourceAccessController() { whitelist = new ArrayList\u0026lt;\u0026gt;(); blacklist = new ArrayList\u0026lt;\u0026gt;(); } public void addToWhitelist(String entity) { whitelist.add(entity); } public void addToBlacklist(String entity) { blacklist.add(entity); } public boolean isAllowed(String entity) { // Check if the entity is in the whitelist and not in the blacklist return whitelist.contains(entity) \u0026amp;\u0026amp; !blacklist.contains(entity); } } public class Main { public static void main(String[] args) { ResourceAccessController accessController = new ResourceAccessController(); // Adding entities to the whitelist accessController.addToWhitelist(\u0026#34;UserA\u0026#34;); accessController.addToWhitelist(\u0026#34;UserB\u0026#34;); // Adding entities to the blacklist accessController.addToBlacklist(\u0026#34;UserC\u0026#34;); // Checking access System.out.println(\u0026#34;UserA allowed?: \u0026#34; + accessController.isAllowed(\u0026#34;UserA\u0026#34;)); // true System.out.println(\u0026#34;UserB allowed?: \u0026#34; + accessController.isAllowed(\u0026#34;UserB\u0026#34;)); // true System.out.println(\u0026#34;UserC allowed?: \u0026#34; + accessController.isAllowed(\u0026#34;UserC\u0026#34;)); // false System.out.println(\u0026#34;UserD allowed?: \u0026#34; + accessController.isAllowed(\u0026#34;UserD\u0026#34;)); // false } } In this example, the ResourceAccessController class has methods to add entities to the allowlist and blocklist. The isAllowed method checks if an entity is in the allowlist and not in the blocklist, thereby determining whether access is allowed. Note that this is a basic example, and in a real-world scenario, you might have more sophisticated access control mechanisms and data structures depending on your requirements.\nConclusion: # With \u0026ldquo;Input Validation\u0026rdquo;, you can ward off many attack vectors on a program and thus help make your software significantly more secure. The end user benefits from a more robust and reliable solution, and the software operator can also expect fewer unpleasant surprises from hackers. All in all, it is an approach that can be implemented anywhere in the application, does not require a lot of implementation effort, and, at the same time, is a cornerstone of security.\nHappy Coding\n","date":"13 December 2023","externalUrl":null,"permalink":"/posts/secure-coding-practices-input-validation/","section":"Posts","summary":"What is - Input Validation? # Input validation is a process used to ensure that the data provided to a system or application meets specific criteria or constraints before it is accepted and processed. The primary goal of input validation is to improve the reliability and security of a system by preventing invalid or malicious data from causing errors or compromising the system’s integrity.\n","title":"Secure Coding Practices - Input Validation","type":"posts"},{"content":"A subdomain takeover is a type of cybersecurity vulnerability that occurs when an attacker gains control of a subdomain of a website or a domain name. This attack can seriously affect the security and functionality of a web application or website. In this explanation, we\u0026rsquo;ll look at subdomain takeovers, how they work, the risks they pose, and how to prevent them.\nIntroduction to subdomains # To understand what a subdomain takeover is, we need to understand the concept of subdomains. In the context of the Internet, a domain name such as \u0026ldquo;example.com\u0026rdquo; serves as the primary address for a website. However, a domain name can be further divided into subdomains, which act as separate sections or branches of the main domain. Subdomains are often used for organisational purposes, categorising different website areas, or providing specific services.\nFor example, consider the subdomains of the abc.com domain:\nwww.abc.com (for the main website) blog.abc.com (for a blog section) shop.abc.com (for an online shop) mail.abc.com (for email services) api.abc.com (for application programming interfaces) These subdomains can be independently configured to provide different content or applications. Although subdomains are part of the same parent domain, they may be hosted on other servers or managed by different teams within the same organisation.\nWhat is a subdomain takeover? # A subdomain takeover occurs when an unauthorised person or organisation gains control of a subdomain of a domain and thereby effectively takes ownership of it. This unauthorised control can lead to various security risks and potential misuse. The key factors contributing to subdomain takeover are misconfigurations, abandoned resources, and external service integrations.\nHere are the main reasons why subdomain takeovers can occur:\nhttps://youtu.be/rhrSC_4neAY\nMisconfigurations: # Web administrators and developers can accidentally misconfigure DNS records or forget to remove DNS records for no more prolonged use subdomains. These misconfigurations can allow an attacker to take control of a subdomain.\nAbandoned resources: # Organisations can create subdomains for specific projects, campaigns, or temporary services and then abandon them later. If an attacker discovers an abandoned subdomain, they can claim it as their own and potentially use it for malicious purposes.\nExternal service integrations: # Many websites integrate third-party services, such as content delivery networks (CDNs), cloud services or hosted applications. These services often require the creation of subdomains that point to the third-party service provider\u0026rsquo;s infrastructure. If the external service becomes vulnerable, attackers can exploit it to take over the associated subdomain.\nHow subdomain takeovers work # Subdomain takeovers result from vulnerabilities due to misconfigured DNS records, abandoned resources, or compromised external services.\nIncorrectly configured DNS entries: # CNAME Records: A common misconfiguration involves CNAME (canonical name) records. A CNAME record maps a subdomain to another domain or subdomain. If a subdomain is incorrectly pointed to a domain controlled by an attacker, the attacker can effectively take over the subdomain.\nWildcard entries: # Wildcard DNS records are used to direct all subdomains of a domain to a common destination. If a subdomain with a wildcard entry is no longer claimed or points to an attacker\u0026rsquo;s server, the attacker can control all subdomains.\nAbandoned resources: # Subdomains created for temporary projects, campaigns, or services may be forgotten or abandoned after their purpose has been served. Attackers can search for and claim such subdomains if they are available for registration or reconfiguration. I want to describe an example I have found several times in projects.\nLet\u0026rsquo;s assume a service is hosted externally. For example, a virtual machine is created and assigned a logical name in Heroku. This typically looks like this: You can choose a name to which the external operator\u0026rsquo;s domain is added. In this example, it is demapp.heroku.com. This service is no longer needed and will be cancelled by the customer. However, there are still entries in the DNS configuration that point to demoapp.heroku.com. This can be, for example, a redirect from your own subdomain in this example, demoapp.abc.com. The attacker now creates an instance on Heroku and gives it the name demoapp , leading to the complete name demoapp.heroku.com. Now, the attacker only has to listen to the ports and analyse which requests arrive at this machine and can build his attack vector based on this. For example, he can set up a corresponding compromised service that responds to these requests.\nCompromised external services: # Some subdomains are created to link to external services or resources provided by third parties. Attackers can tamper with the associated subdomains if these external services are compromised or misconfigured.\nIn these scenarios, the attacker effectively takes control of the subdomain by configuring the DNS records or exploiting vulnerabilities. Once they control the subdomain, they can host malicious content, launch phishing attacks, or commit other forms of cybercrime.\nIn these scenarios, the attacker effectively takes control of the subdomain by configuring the DNS records or exploiting vulnerabilities. Once they control the subdomain, they can host malicious content, launch phishing attacks, or commit other forms of cybercrime.\nRisks and consequences of subdomain takeovers # Subdomain takeovers pose significant risks to websites\u0026rsquo; and web applications\u0026rsquo; security, reputation and functionality. The consequences of a successful subdomain takeover can be severe and far-reaching, including:\nData theft: # Attackers can use subdomains to steal sensitive data such as user credentials, personal information, or financial data.\nPhishing attacks: # Subdomain takeovers are often used in phishing campaigns. Attackers can create convincing fake login pages or legitimate content to trick users into revealing their credentials.\nMalware distribution: # Malicious files or software can be hosted on the compromised subdomain, which can then be used to distribute malware to unsuspecting visitors.\nDenylisting: # Search engines and email providers can denylist the parent domain if they detect malicious activity on a subdomain. This may impact website visibility and email communications.\nBrand damage: # A subdomain takeover can damage an organisation or website\u0026rsquo;s reputation and lose trust among users and customers.\nFinancial loss: # Depending on the severity of the attack, a company could face financial losses, lawsuits, and fines.\nLoss of control: # Once an attacker has control of a subdomain, it becomes difficult for the rightful owner to regain control and limit the damage.\nLegal consequences: # Unauthorised access to or control over subdomains can result in legal consequences and penalties for the attacker.\nPractical examples # To illustrate the real-world impact of subdomain takeovers, let\u0026rsquo;s look at a few notable cases:\nUber: # In 2015, researchers discovered a subdomain takeover vulnerability in Uber\u0026rsquo;s system. The ride-hailing company had a subdomain called riders.uber.com that was vulnerable to a takeover. Attackers may have used this subdomain to launch phishing attacks against Uber users. Uber immediately addressed the issue after being informed by researchers.\nGitHub pages: # GitHub Pages is a popular platform for hosting websites and documentation. In 2017, a researcher found that GitHub Page\u0026rsquo;s subdomains were vulnerable to a takeover when users deleted their repositories but left the associated DNS records intact. Attackers could then claim the subdomains and host malicious content. GitHub has since taken measures to prevent such takeovers.\nShopify: # Shopify, a central e-commerce platform, faced a subdomain adoption issue in 2019. An attacker discovered that he could claim abandoned Shopify store subdomains and potentially use them for fraud. Shopify has fixed the vulnerability and improved its security measures.\nThese cases show that subdomain takeovers can affect both small and large organisations. Timely identification and remediation of such vulnerabilities is critical to maintaining the security and trustworthiness of online services.\nPrevent subdomain takeovers # Preventing subdomain takeovers requires a combination of proactive measures and best practices. Here are steps companies can take to mitigate the risk of subdomain takeovers:\nRegular DNS Audits: # Perform routine DNS record checks to identify misconfigurations or abandoned subdomains. Remove unnecessary DNS records and update records for active subdomains.\nBest practices for CNAME records: # When using CNAME records, be careful where they point. Make sure you trust the target and check CNAME configurations regularly.\nWildcard DNS records: # Use wildcard DNS records carefully. Avoid directing all subdomains to a single destination, as this can create a single point of failure.\nSubdomain Registration Guidelines: # Implement clear guidelines for subdomain creation and management. Ensure that subdomains are registered with valid entities within the organisation and follow a consistent naming convention.\nAutomated scanning: # Use automated scanning tools to identify subdomains at risk of takeover. These tools can help identify misconfigurations and vulnerabilities.\nSecurity of external services: # If you use external services requiring subdomains, ensure those services are secure. Regularly monitor and audit the security of these services to prevent potential vulnerabilities.\nDNS rate limit: # Implement rate limiting for DNS requests to protect against reconnaissance attacks from potential attackers.\nAccess control: # Restrict access to DNS configuration and ensure only authorised personnel can change DNS records.\nVulnerability disclosure programs: # Encourage security researchers and users to report subdomain takeover vulnerabilities responsibly. Establish a process for receiving and processing such reports.\nMonitor subdomain activity: # Monitor subdomain activity and set up alerts for suspicious or unauthorised changes to DNS records.\nConclusion # Subdomain takeovers represent a widespread cybersecurity threat and can significantly impact website owners and users. These takeovers are often due to misconfigured DNS records, abandoned resources, or compromised external services.\nTo protect against subdomain takeovers, companies/teams must proactively monitor and secure their subdomains. This includes regular DNS audits, best practices for managing DNS records, and a strong awareness of potential vulnerabilities in external services.\nHappy Coding\nSven\n","date":"20 November 2023","externalUrl":null,"permalink":"/posts/infection-method-sub-domain-takeover/","section":"Posts","summary":"A subdomain takeover is a type of cybersecurity vulnerability that occurs when an attacker gains control of a subdomain of a website or a domain name. This attack can seriously affect the security and functionality of a web application or website. In this explanation, we’ll look at subdomain takeovers, how they work, the risks they pose, and how to prevent them.\n","title":"Infection Method - Sub-Domain Takeover","type":"posts"},{"content":"","date":"20 November 2023","externalUrl":null,"permalink":"/tags/subdomain-takeover/","section":"Tags","summary":"","title":"Subdomain Takeover","type":"tags"},{"content":"","date":"10 November 2023","externalUrl":null,"permalink":"/tags/domain-takeover/","section":"Tags","summary":"","title":"Domain Takeover","type":"tags"},{"content":"In this post, we will look at another method of infection. These are the attack vectors via domain names. This can happen at the main level, i.e. the domain itself, or via sub-domains. But what exactly is a domain takeover attack?\nA domain takeover is a cyberattack when an attacker gains control of a domain name owned by another person or organization. This can have severe consequences as the attacker can use the domain for malicious purposes, such as spreading malware, phishing, or taking control of a company\u0026rsquo;s online presence.\nBelow, we will look at different ways in which such a takeover can take place:\n1. Expired domain : # A common possibility of domain takeover is that the owner of a domain forgets to renew it. When a domain registration expires, it may be available for purchase by anyone. Attackers can monitor the expiration of valuable domains and quickly register them as soon as they become available. If you have set up automatic renewal with your domain registrar, this should not happen. But sometimes companies (or people) get rid of old domains. This could be because the acquired company is now fully integrated and the domain is no longer needed, or because a project has expired, or or or. The problem, then, is that these domains may still be included in the internal configurations and firewall rules. The attacker then exploits the extended privileges typically intended for their own systems. If the old domains are now for sale, the attacker has purchased them regularly, and the own configurations still need to be updated, an attack can be carried out.\nhttps://youtu.be/I5ZX5yX7rOI\n2. Domain-Hijacking : # In some cases, attackers can use social engineering techniques to trick domain registrars or Domain Name System (DNS) providers into giving them domain control. This could mean impersonating the domain owner or providing false information to the domain registrar. Here, however, you have to think in different directions. Here is an example of the domain used for a reverse attack on a Maven repository. An open-source project named ABC does not have its associated domain secured. The Maven artefacts are stored in the central Maven Central, for example, under “org.abc”. The attacker now registered the domain abc.org and then contacted the operator of Maven Central, saying that he needed access to his Maven repository. He has lost his access to data. The operator of Maven-Central then asks the applicant to store a TXT entry with a specific code (in this case, it is the ticket number of the request) in the DNS configuration. Once this code is available via DNS, rights to the repository are granted. The attacker can now deposit his Maven artefacts in the repository and thus make them generally available.\n3. DNS-Misconfiguration : # Sometimes, domain takeovers occur due to misconfigurations in DNS settings. Attackers can exploit these misconfigurations to gain control of the domain or subdomains and thereby redirect traffic to malicious servers.\n3.1 What is a critical DNS misconfiguration? # A critical Domain Name System (DNS) misconfiguration is an error or error in the configuration of a DNS system that can have severe consequences for the availability, security, and functionality of a domain or network. DNS is a crucial Internet component that translates human-readable domain names (like example.com) into IP addresses that computers use to identify and communicate with each other. When DNS misconfiguration occurs, it can lead to various problems, including:\n1. Service interruption : # A misconfigured DNS record can cause service outages, making a website or other online services inaccessible.\n2. Vulnerabilities : # Misconfigurations can lead to security vulnerabilities, e.g. B., to reveal sensitive information, enable unauthorized access, or enable DNS-related attacks such as DNS cache poisoning or DNS spoofing.\n3. Data Loss : # Incorrect DNS configurations can lead to data loss as changes to DNS records can result in email misdirection or loss of crucial domain-related information.\n4. Performance Issues : # Suboptimal DNS configurations can slow down domain name resolution and cause delays in website loading or other network activity.\n5. Traffic Diversion : # DNS misconfigurations can inadvertently direct traffic to the wrong IP addresses or locations, potentially leading to data leaks, man-in-the-middle attacks, or other unintended consequences.\n6. Domain-Hijacking : # This is exactly the case of domain takeover considered here, in which unauthorized persons gain control of a domain and the services associated with it. Common examples of critical DNS misconfigurations include errors in DNS records (e.g. A, CNAME, MX, TXT records), incorrect IP address assignments, outdated or expired DNS information, and unauthorized changes to DNS settings .\n4. Phishing/Theft Of Credentials : # Attackers can also use phishing attacks to trick domain owners or those with administrative access to domain management accounts into revealing their credentials. Once they have the credentials, they can log in and take control of the domain.\n5. Subdomain-TakeOver : # A subdomain takeover occurs when an unauthorized person or organization gains control of a subdomain of a domain and thereby effectively takes ownership of it. This unauthorized control can lead to various security risks and potential misuse. The key factors contributing to subdomain takeover are misconfigurations, abandoned resources, and external service integrations.\n6. DNS-Cache-Poisoning : # In some cases, attackers may attempt to poison DNS caches, causing them to resolve a legitimate domain into a malicious IP address. This can lead to a temporary domain takeover as users are redirected to the attacker\u0026rsquo;s server rather than the intended website.\nOnce the attacker gains control of a domain, they can use it for various malicious purposes, such as hosting fake websites for phishing attacks, distributing malware, or intercepting communications. This can damage the domain owner\u0026rsquo;s reputation, jeopardize user security and privacy, and result in legal and financial consequences for the legitimate domain owner.\nAn example from history: # A notable historical example of a domain takeover is the case of the Syrian Electronic Army (SEA) in 2013. The SEA was a group of hacktivists who supported the Syrian government and targeted various websites, social media accounts and domains to promote their political agenda. One of the most high-profile incidents involved the takeover of the Twitter accounts and the domain of several well-known media organizations.\nIn April 2013, SEA compromised the domain registration account of MarkMonitor, a domain registrar and provider of brand protection services. Using stolen credentials or other means, SEA accessed the New York Times\u0026rsquo; domain registration records. They changed the domain\u0026rsquo;s DNS records, redirecting traffic from the New York Times website to a server controlled by the SEA. As a result, visitors to The New York Times website were greeted with a message from SEA instead of the expected news content.\nThis domain takeover disrupted the New York Times\u0026rsquo; online operations and raised concerns about the security of domain registrars. The incident highlighted the importance of protecting domain management accounts and the potential impact of domain takeovers on well-known media organizations and their readership.\nHow do you protect yourself from these attacks? # To protect yourself from domain takeovers, it is essential for domain owners to:\n1. Keep your domain registrations current and renew them on time.\n2. Use strong authentication and authorization mechanisms for domain management accounts.\n3. Regularly monitor and check your DNS configurations for misconfigurations.\n4. Educate your team about the risks of social engineering and phishing attacks.\n5. Use domain security services and technologies to detect and prevent unauthorized changes to domain settings.\nThis is not an exhaustive list, but it is a good start to preventing the most common attack vectors.\nConclusion: # We have seen many different attack vectors that can lead to a loss of control of the (sub)domain. These attacks are still prevalent and are successfully used even on large companies. Unfortunately, many small companies have exactly these vulnerabilities of abandoned resources in the sub-domain area. I recommend that every project member pay close attention to what else can be actively found in the configurations of the firewalls and DNS, even when it comes to testing and development infrastructure.\nHappy Coding\nSven\n","date":"10 November 2023","externalUrl":null,"permalink":"/posts/infection-method-domain-takeover/","section":"Posts","summary":"In this post, we will look at another method of infection. These are the attack vectors via domain names. This can happen at the main level, i.e. the domain itself, or via sub-domains. But what exactly is a domain takeover attack?\n","title":"Infection Method - Domain Takeover","type":"posts"},{"content":"I will introduce you to the serializer from the EclipseStore project and show you how to use it to take advantage of a new type of serialization.\nSince I learned Java over 20 years ago, I wanted to have a simple solution to serialize Java-Object-Graphs, but without the serialization security and performance issues Java brought us. It should be doable like the following…\nbyte[] data = serializer.serialize(objectGraph); Node objectGraphDeserialized = serializer.deserialize(data); If you want to know how this is doable with the new Open-Source Project EclipseSerializer? You are right here.\nBefore we look at the Open-Source project EclipseStore Serializer, I want to recap a bit of the challenges coming from the Java Serialization itself. This will be the background information to see how powerful the project is.\nJava Serialization in a nutshell # Java Serialization is a mechanism provided by the Java programming language that allows you to convert the state of an object into a byte stream. This byte stream can be easily stored in a file, sent over a network, or otherwise persisted. Later, you can deserialize the byte stream to reconstruct the original object, effectively saving and restoring the object\u0026rsquo;s state.\nHere are some key points about Java Serialization:\nSerializable Interface: To make a Java class serializable, it needs to implement the Serializable interface. This interface doesn\u0026rsquo;t have any methods; it acts as a marker interface to indicate that the objects of this class can be serialized.\nimport java.io.Serializable; public class MyClass implements Serializable { // class members and methods } Serialization Process: To serialize an object, you typically use ObjectOutputStream. You create an instance of this class and write your object to it. For example:\ntry (FileOutputStream fileOut = new FileOutputStream(\u0026#34;object.ser\u0026#34;); ObjectOutputStream out = new ObjectOutputStream(fileOut)) { MyClass obj = new MyClass(); out.writeObject(obj); } catch (IOException e) { e.printStackTrace(); } Deserialization Process: To deserialize an object, you use ObjectInputStream. You read the byte stream from a file or network and then use ObjectInputStream to recreate the object.\ntry (FileInputStream fileIn = new FileInputStream(\u0026#34;object.ser\u0026#34;); ObjectInputStream in = new ObjectInputStream(fileIn)) { MyClass obj = (MyClass) in.readObject(); // Now \u0026#39;obj\u0026#39; contains the deserialized object } catch (IOException | ClassNotFoundException e) { e.printStackTrace(); } Versioning Considerations: If you change a serialised class\u0026rsquo;s structure (e.g., adding or removing fields or changing their types), the deserialization of old serialized objects may fail. Java provides mechanisms like serialVersionUID to help with versioning and compatibility.\nSecurity Considerations: Serialization can be a security risk, especially if you deserialize data from untrusted sources. Malicious code can be executed during deserialization. To mitigate this risk, you should carefully validate and sanitize any data you deserialize or consider alternative serialization mechanisms like JSON or XML.\nCustom Serialization: In your class, you can customize the serialization and deserialization process by providing writeObject and readObject methods. These methods allow you to control how the object\u0026rsquo;s state is written to and read from the byte stream.\nIn summary, Java Serialization is a valuable feature for persisting objects and sending them across a network. However, it comes with some challenges related to versioning and security, so it should be used cautiously, especially when dealing with untrusted data sources.\nWhat are the security issues # Java Serialization can introduce several security issues, particularly when deserializing data from untrusted sources. Here are some of the security concerns associated with Java Serialization:\nRemote Code Execution: One of the most significant security risks with Java Serialization is the potential for remote code execution. When you deserialize an object, the Java runtime system can execute arbitrary code contained within the serialized data. Attackers can exploit this to execute malicious code on the target system. This vulnerability can lead to serious security breaches.\nDenial of Service (DoS): An attacker can create a serialized object with a large size, causing excessive memory consumption and potentially leading to a denial of service attack. Deserializing large objects can consume significant CPU and memory resources, slowing down or crashing the application.\nData Tampering: Serialized data can be tampered with during transmission or storage. Attackers can modify the serialized byte stream to alter the state of the deserialized object or introduce vulnerabilities.\nInsecure Deserialization: Deserializing untrusted data without proper validation can lead to security issues. For example, if a class that performs sensitive operations is deserialized from untrusted input, an attacker can manipulate the object\u0026rsquo;s state to perform unauthorized actions.\nInformation Disclosure: When objects are serialized, sensitive information may be included in the serialized form. If this data is not adequately protected or encrypted, an attacker may gain access to sensitive information.\nHow To Mitigate Serialization Issues # To mitigate these security issues, consider the following best practices:\nAvoid Deserializing Untrusted Data: If possible, avoid deserializing data from untrusted sources altogether. Instead, use safer data interchange formats like JSON or XML for untrusted data. (Or use the EclipseSerializer ;-) )\nImplement Input Validation: When deserializing data, validate and sanitize the input to ensure it adheres to expected data structures and doesn\u0026rsquo;t contain unexpected or malicious data.\nUse Security Managers: Java\u0026rsquo;s Security Manager can be used to restrict the permissions and actions of deserialized code. However, it\u0026rsquo;s important to note that Security Managers have been deprecated in newer versions of Java.\nWhitelist Classes: Limit the classes that can be deserialized to a predefined set of trusted classes. This can help prevent the deserialization of arbitrary and potentially malicious classes.\nVersioning and Compatibility: Be cautious when making changes to serialized classes. Use serialVersionUID to manage versioning and compatibility between different versions of serialized objects.\nSecurity Libraries: Consider using third-party libraries like Apache Commons Collections or OWASP Java Serialization Security (Java-Serial-Killer) to help mitigate known vulnerabilities and prevent common attacks.\nIn summary, Java Serialization can introduce serious security risks, especially when dealing with untrusted data. It\u0026rsquo;s essential to take precautions, validate inputs, and consider alternative serialization methods or libraries to enhance security. Additionally, keeping your Java runtime environment up to date is crucial, as newer versions of Java may include security improvements and fixes for known vulnerabilities.\nWhy is JSON or XML not the perfect solution for the JVM? # Many papers and lectures recommend circumventing the security risks of serialization by using XML or JSON. This is a structured representation of the data that is to be transferred. There are also security problems, but I will address these in a separate article. However, what should be addressed are two things. First, the data must be converted into a text representation. This usually requires more data volume than with a pure binary model. In addition, data such as the binary data of images must be recoded so that only printable or UTF-8 characters can be transmitted. This process requires a lot of time and usually a lot of memory when transforming it into XML and back from XML into the original format.\nThe second point that causes problems in most cases is the data structure. In XML and JSON, object references can only be stored in a more manageable manner. This makes processing many times more complicated, slower and more resource-intensive. Even though many solid solutions can be used to convert Java objects into XML or JSON, I recommend looking for new approaches occasionally.\nEclipseStore - Serializer - Practical Part # Now, let\u0026rsquo;s get to the practical stuff in this article. The dependency is needed first. To do this, we add the following instructions to the pom.xml. The first release was prepared when writing this article, and a SNAPSHOT version (1.0.0-SNAPSHOT) was available from the repositories. In this case, you still have to use the SNAPSHOT repositories (https://oss.sonatype.org/content/repositories/snapshots)\nDefinition inside the pom.xml.\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.eclipse.serializer\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;serializer\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;{maven-version-number}\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; The rest will happen quickly once we\u0026rsquo;re ready and have fetched the dependency. For the first test, we created a class called Node. Each Node can have a right and a left child. With this, we can create a tree.\nprivate String id; private Node leftNode; private Node rightNode; As an example, I created the following construct and then serialized and de-serialized it once using the serializer.\nNode rootNode = new Node(\u0026#34;rootNode\u0026#34;); Node leftChildLev01 = new Node(\u0026#34;Root-L\u0026#34;); Node rightChildLev01 = new Node(\u0026#34;Root-R\u0026#34;); leftChildLev01.addLeft(new Node(\u0026#34;Root-L-R\u0026#34;)); leftChildLev01.addLeft(new Node(\u0026#34;Root-L-L\u0026#34;)); rightChildLev01.addLeft(new Node(\u0026#34;Root-R-L\u0026#34;)); rightChildLev01.addRight(new Node(\u0026#34;Root-R-R\u0026#34;)); rootNode.addLeft(leftChildLev01); rootNode.addRight(rightChildLev01); Serializer\u0026lt;byte[]\u0026gt; serializer = Serializer.Bytes(); byte[] data = serializer.serialize(rootNode); Node rootNodeDeserialized = serializer.deserialize(data); System.out.println(rootNode.toString()); System.out.println(\u0026#34; ========== \u0026#34;); System.out.println(rootNodeDeserialized.toString()); Now, let\u0026rsquo;s see whether this also works with a Java object graph. To do this, the class representing the node is changed so that a father node can also be defined. Cycles can now be set up.\npublic class GraphNode { private String id; private GraphNode parent; private List\u0026lt;GraphNode\u0026gt; childGraphNodes = new ArrayList\u0026lt;\u0026gt;(); We take the graph listed here as an example.\nGraphNode rootNode = new GraphNode(\u0026#34;rootNode\u0026#34;); GraphNode child01Lev01 = new GraphNode(\u0026#34;child01Lev01\u0026#34;); GraphNode child02Lev01 = new GraphNode(\u0026#34;child02Lev01\u0026#34;); rootNode.addChildGraphNode(child01Lev01); rootNode.addChildGraphNode(child02Lev01); child01Lev01.setParent(rootNode); child02Lev01.setParent(rootNode); GraphNode child01Lev02 = new GraphNode(\u0026#34;child01Lev02\u0026#34;); GraphNode child02Lev02 = new GraphNode(\u0026#34;child02Lev02\u0026#34;); child01Lev01.addChildGraphNode(child01Lev02); child01Lev01.addChildGraphNode(child02Lev02); child01Lev02.setParent(child01Lev01); child01Lev02.setParent(child01Lev01); GraphNode child01Lev03 = new GraphNode(\u0026#34;child01Lev03\u0026#34;); GraphNode child02Lev03 = new GraphNode(\u0026#34;child02Lev03\u0026#34;); child01Lev03.setParent(child02Lev01); child02Lev03.setParent(child02Lev01); child02Lev01.addChildGraphNode(child01Lev03); child02Lev01.addChildGraphNode(child02Lev03); //creating cycles rootNode.addChildGraphNode(child01Lev03); rootNode.setParent(child02Lev03); This graph is also processed without any problems, without the cycles causing any issues.\nSerializer\u0026lt;byte[]\u0026gt; serializer = Serializer.Bytes(); byte[] data = serializer.serialize(rootNode); GraphNode rootNodeDeserialized = serializer.deserialize(data); You can make the examples even more complex and try out the subtleties of inheritance. New data types from JDK17 are also supported. This means I have a potent tool to handle various tasks. For example, one use can be found in another Eclipse project called EclipseStore. A persistence mechanism is provided here based on this serialization. But your own small projects can also benefit from this. I will show how quickly this can be integrated into a residual service.\nBuilding A Simple REST Service # If you want to create a simple REST service for transferring byte streams in Java without using Spring Boot, you can use the Java SE API and the HttpServer class from the com.sun.net.httpserver package, which allows you to create an HTTP server.\nWe create an HTTP server on port 8080 and define a context for handling requests to \u0026ldquo;/api/bytestream.\u0026rdquo; The ByteStreamHandler class handles both POST requests for uploading byte streams and GET requests for downloading byte streams. For POST requests, it reads the incoming byte stream, processes it as needed, and sends a response. For GET requests, it sends a predefined byte stream as a response. Remember that this is a simple example, and you can expand upon it to handle more complex use cases and error handling as needed for your specific application. Also, note that the com.sun.net.httpserver package is part of the JDK, but it may not be available in all Java distributions.\npublic class ByteStreamHandler implements HttpHandler { @Override public void handle(HttpExchange exchange) throws IOException { String requestMethod = exchange.getRequestMethod(); if (requestMethod.equalsIgnoreCase(\u0026#34;POST\u0026#34;)) { // Handle POST requests for uploading byte streams handleUpload(exchange); } else if (requestMethod.equalsIgnoreCase(\u0026#34;GET\u0026#34;)) { // Handle GET requests for downloading byte streams handleDownload(exchange); } } private void handleUpload(HttpExchange exchange) throws IOException { // Get the input stream from the request InputStream inputStream = exchange.getRequestBody(); // Read the byte stream and process it as needed ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); byte[] buffer = new byte[1024]; int bytesRead; while ((bytesRead = inputStream.read(buffer)) != -1) { byteArrayOutputStream.write(buffer, 0, bytesRead); } // Process the uploaded byte stream // (e.g., save to a file or perform other actions) byte[] data = byteArrayOutputStream.toByteArray(); // Do something with \u0026#39;data\u0026#39; Serializer\u0026lt;byte[]\u0026gt; serializer = Serializer.Bytes(); GraphNode deserialized = serializer.deserialize(data); //process the data System.out.println(\u0026#34;deserialized = \u0026#34; + deserialized); // Send a response (you can customize this) String response = \u0026#34;Byte stream uploaded successfully.\u0026#34;; exchange.sendResponseHeaders(200, response.length()); OutputStream os = exchange.getResponseBody(); os.write(response.getBytes()); os.close(); } private void handleDownload(HttpExchange exchange) throws IOException { // Simulate generating and sending a byte stream as a response String response = \u0026#34;Hello, Byte Stream!\u0026#34;; Serializer\u0026lt;byte[]\u0026gt; serializer = Serializer.Bytes(); byte[] data = serializer.serialize(response.getBytes()); exchange.sendResponseHeaders(200, data.length); OutputStream os = exchange.getResponseBody(); os.write(data); os.close(); } } Conclusion: # We\u0026rsquo;ve looked at the typical problems with Java\u0026rsquo;s original serialization and how cumbersome the implementation is to use. The detour via JSON and XML is unnecessary when communicating from JVM to JVM using the open-source Eclipse Serializer project. There are no restrictions when it comes to modelling a graph, as not only are the current new data types up to and including JDK17 already processed, but cycles within the graph are also no problem.\nUsing the Serializable interface is also unnecessary and does not influence processing. The easy handling allows it to be used even in tiny projects such as the REST service shown here using on-board JDK resources. A larger project that uses the serializer is the open-source project EclipseStore. A high-performance persistence mechanism for the JVM is offered here.\nHappy Coding\nSven\n","date":"9 October 2023","externalUrl":null,"permalink":"/posts/eclipsestore-high-performance-serializer/","section":"Posts","summary":"I will introduce you to the serializer from the EclipseStore project and show you how to use it to take advantage of a new type of serialization.\nSince I learned Java over 20 years ago, I wanted to have a simple solution to serialize Java-Object-Graphs, but without the serialization security and performance issues Java brought us. It should be doable like the following…\n","title":"EclipseStore High-Performance-Serializer","type":"posts"},{"content":"How to store complex data structures using EclipseStore? Are there any restrictions? How can you work more efficiently on such structures? We will now get to the bottom of these questions here.\nIn the first part of my series, I showed how to prepare EclipseStore for use in a project. We also initialized the StorageManager and saved, modified and deleted the first data. But what about more complex structures? Can you use inheritance? To do this, we will now create a small class model.\nFirst, let\u0026rsquo;s look at what inheritance looks like. To do this, we take an interface called BaseInterfaceA , an implementation BaseClassA and a derivative LevelOneA. We will now try to save this and see how it behaves depending on the input when saving.\nHere is the listing of the classes and interfaces involved. Since we already saw in the first part that the implementation variant has no influence, I will use a standard implementation regarding getters and setters or the constructors.\npublic interface BaseInterfaceA { public default String allUpperCase(String value){ return value.toUpperCase(); } } public class BaseClassA implements BaseInterfaceA { private String valueBaseA; @Override public String toString() { return \u0026#34;BaseClassA{\u0026#34; + \u0026#34;valueBaseA=\u0026#39;\u0026#34; + valueBaseA + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#39;}\u0026#39;; } @Override public boolean equals(Object o) { if (this == o) return true; if (o == null || getClass() != o.getClass()) return false; BaseClassA that = (BaseClassA) o; return Objects.equals(valueBaseA, that.valueBaseA); } @Override public int hashCode() { return Objects.hash(valueBaseA); } public String getValueBaseA() { return valueBaseA; } public void setValueBaseA(String valueBaseA) { this.valueBaseA = valueBaseA; } } public class LevelOneA extends BaseClassA { private String valueOneA; public LevelOneA(String valueOneA) { this.valueOneA = valueOneA; } @Override public boolean equals(Object o) { if (this == o) return true; if (o == null || getClass() != o.getClass()) return false; if (!super.equals(o)) return false; LevelOneA levelOneA = (LevelOneA) o; return Objects.equals(valueOneA, levelOneA.valueOneA); } @Override public int hashCode() { return Objects.hash(super.hashCode(), valueOneA); } public String getValueOneA() { return valueOneA; } public void setValueOneA(String valueOneA) { this.valueOneA = valueOneA; } @Override public String toString() { return \u0026#34;LevelOneA{\u0026#34; + \u0026#34;valueOneA=\u0026#39;\u0026#34; + valueOneA + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#39;}\u0026#39;; } } Case I: Direct saving of the respective classes # The entities are packed directly into a list and saved in the first case. As we can see, there are no special features here. This is the same case as the first article. The output of the printElements method is exactly what we expected.\npublic static void main(String[] args) { EmbeddedStorageManager storageManager = EmbeddedStorage.start(); storageManager.setRoot(new ArrayList\u0026lt;LevelOneA\u0026gt;()); storageManager.storeRoot(); List\u0026lt;LevelOneA\u0026gt; root = (List\u0026lt;LevelOneA\u0026gt;) storageManager.root(); root.add(new LevelOneA(\u0026#34;levelOneA - 01\u0026#34;)); root.add(new LevelOneA(\u0026#34;levelOneA - 02\u0026#34;)); root.add(new LevelOneA(\u0026#34;levelOneA - 03\u0026#34;)); storageManager.storeRoot(); storageManager.shutdown(); storageManager = EmbeddedStorage.start(); List\u0026lt;LevelOneA\u0026gt; rootLoaded = (List\u0026lt;LevelOneA\u0026gt;) storageManager.root(); rootLoaded.forEach(System.out::println); System.out.println(\u0026#34; ========== \u0026#34;); storageManager.shutdown(); } console output:\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 01\u0026rsquo;}\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 02\u0026rsquo;}\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 03\u0026rsquo;}\n==========\nCase II: Save as a base class # Now, let\u0026rsquo;s consider the case where a class is passed to the StorageManager for storage as one of its base classes. In this case, the LevelOneA class is passed as the base class BaseClassA. It should be noted that this primary type was also used when defining the root list.\nEmbeddedStorageManager storageManager = EmbeddedStorage.start(); storageManager.setRoot(new ArrayList\u0026lt;BaseClassA\u0026gt;()); storageManager.storeRoot(); List\u0026lt;BaseClassA\u0026gt; root = (List\u0026lt;BaseClassA\u0026gt;) storageManager.root(); root.add(new LevelOneA(\u0026#34;levelOneA - 01\u0026#34;)); root.add(new LevelOneA(\u0026#34;levelOneA - 02\u0026#34;)); root.add(new LevelOneA(\u0026#34;levelOneA - 03\u0026#34;)); storageManager.storeRoot(); storageManager.shutdown(); storageManager = EmbeddedStorage.start(); List\u0026lt;BaseClassA\u0026gt; rootLoaded = (List\u0026lt;BaseClassA\u0026gt;) storageManager.root(); rootLoaded.forEach(System.out::println); System.out.println(\u0026#34; ========== \u0026#34;); storageManager.shutdown(); The output again meets our expectations.\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 01\u0026rsquo;}\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 02\u0026rsquo;}\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 03\u0026rsquo;}\n==========\nCase II: Saving an implementation as an interface # Now let\u0026rsquo;s move on to the case in which we proceed via the interface. There are no problems here either. Everything behaves as expected.\nEmbeddedStorageManager storageManager = EmbeddedStorage.start(); storageManager.setRoot(new ArrayList\u0026lt;BaseInterfaceA\u0026gt;()); storageManager.storeRoot(); List\u0026lt;BaseInterfaceA\u0026gt; root = (List\u0026lt;BaseInterfaceA\u0026gt;) storageManager.root(); root.add(new LevelOneA(\u0026#34;levelOneA - 01\u0026#34;)); root.add(new LevelOneA(\u0026#34;levelOneA - 02\u0026#34;)); root.add(new LevelOneA(\u0026#34;levelOneA - 03\u0026#34;)); storageManager.storeRoot(); storageManager.shutdown(); storageManager = EmbeddedStorage.start(); List\u0026lt;BaseInterfaceA\u0026gt; rootLoaded = (List\u0026lt;BaseInterfaceA\u0026gt;) storageManager.root(); rootLoaded.forEach(System.out::println); System.out.println(\u0026#34; ========== \u0026#34;); storageManager.shutdown(); The output on the console is still unchanged.\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 01\u0026rsquo;}\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 02\u0026rsquo;}\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 03\u0026rsquo;}\n==========\nCase IV: Saving a mixed list via a common interface # Now if we define a list of type BaseInterfaceA as root and then add instances of different implementations. How does EclipseStore behave during the save process? To make a long story short, it works as intended. All instances are stored neatly after deployment. So we have no loss of information.\nThe console output will then look like this.\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 01\u0026rsquo;}\nBaseClassA{valueBaseA=\u0026lsquo;BaseClassA - 02\u0026rsquo;}\nLevelOneA{valueOneA=\u0026lsquo;levelOneA - 03\u0026rsquo;}\n==========\nStoring lists, trees and graphs # Now that we\u0026rsquo;ve looked at whether inheritance is supported as much as we hoped, we\u0026rsquo;re getting to the point where we can start thinking about the data structures themselves. We have seen that simple entities and lists of entities are relatively easy for EclipseStore. Now, let\u0026rsquo;s go one step further and look at trees and graphs.\nStoring of trees # In computer science, we understand a tree to be a data structure that can have branches but does not contain any cycles. This makes them a special form of graphs, just as lists are a special form of trees. So, let\u0026rsquo;s come to a tree implementation in which a node can have two child nodes. Of course, you can also build this structure with n child nodes, but this will not bring any added value in our case. If we create such a tree and give the root node to the StorageManager, we can save this tree without any further action. Modifying individual elements also works as usual. You can also change the desired segment and save on this or any higher-level piece.\nEmbeddedStorageManager storageManager = EmbeddedStorage.start(); Node rootNode = new Node(\u0026#34;rootNode\u0026#34;); storageManager.setRoot(rootNode); storageManager.storeRoot(); Node leftChildLev01 = new Node(\u0026#34;Root-L\u0026#34;); Node rightChildLev01 = new Node(\u0026#34;Root-R\u0026#34;); leftChildLev01.addLeft(new Node(\u0026#34;Root-L-R\u0026#34;)); leftChildLev01.addLeft(new Node(\u0026#34;Root-L-L\u0026#34;)); rightChildLev01.addLeft(new Node(\u0026#34;Root-R-L\u0026#34;)); rightChildLev01.addRight(new Node(\u0026#34;Root-R-R\u0026#34;)); rootNode.addLeft(leftChildLev01); rootNode.addRight(rightChildLev01); storageManager.storeRoot(); storageManager.shutdown(); storageManager = EmbeddedStorage.start(); Node rootLoaded = (Node) storageManager.root(); System.out.println(rootLoaded.toString()); System.out.println(\u0026#34; ========== \u0026#34;); storageManager.shutdown(); Output to the console (subsequently formatted for readability):\nNode{id=\u0026lsquo;rootNode\u0026rsquo;,\nleftNode=Node{id=\u0026lsquo;Root-L\u0026rsquo;,\nleftNode=Node{id=\u0026lsquo;Root-L-L\u0026rsquo;,\nleftNode=null,\nrightNode=null\n},\nrightNode=null\n},\nrightNode=Node{id=\u0026lsquo;Root-R\u0026rsquo;,\nleftNode=Node{id=\u0026lsquo;Root-R-L\u0026rsquo;,\nleftNode=null,\nrightNode=null\n},\nrightNode=Node{id=\u0026lsquo;Root-R-R\u0026rsquo;,\nleftNode=null,\nrightNode=null\n}\n}\n}\nStoring graphs # Unfortunately, you don\u0026rsquo;t just have to deal with lists and trees. Complex data models often have cycles. This can come about, for example, through bidirectional relationships. Unwanted cycles can also arise in a model if it grows over time and is repeatedly expanded. Whether these cycles are tightly or loosely coupled plays a minor role. The following example creates a chart that contains multiple cycles. Can the diagram then be saved, modified and loaded? Does EclipseStore recognize these structures and can resolve or break the loops?\nIn this example, the graph consists of nodes of the GraphNode class. This class contains an attribute of type String to store an ID. There is also a reference to the parent node and a list of child nodes. This means you can now create any nested chart you want.\nIn this case, the graphic looks like this.\nGraphNode rootNode = new GraphNode(\u0026#34;rootNode\u0026#34;); GraphNode child01Lev01 = new GraphNode(\u0026#34;child01Lev01\u0026#34;); GraphNode child02Lev01 = new GraphNode(\u0026#34;child02Lev01\u0026#34;); rootNode.addChildGraphNode(child01Lev01); rootNode.addChildGraphNode(child02Lev01); child01Lev01.setParent(rootNode); child02Lev01.setParent(rootNode); GraphNode child01Lev02 = new GraphNode(\u0026#34;child01Lev02\u0026#34;); GraphNode child02Lev02 = new GraphNode(\u0026#34;child02Lev02\u0026#34;); child01Lev01.addChildGraphNode(child01Lev02); child01Lev01.addChildGraphNode(child02Lev02); child01Lev02.setParent(child01Lev01); child01Lev02.setParent(child01Lev01); GraphNode child01Lev03 = new GraphNode(\u0026#34;child01Lev03\u0026#34;); GraphNode child02Lev03 = new GraphNode(\u0026#34;child02Lev03\u0026#34;); child01Lev03.setParent(child02Lev01); child02Lev03.setParent(child02Lev01); child02Lev01.addChildGraphNode(child01Lev03); child02Lev01.addChildGraphNode(child02Lev03); //creating cycles rootNode.addChildGraphNode(child01Lev03); rootNode.setParent(child02Lev03); EmbeddedStorageManager storageManager = EmbeddedStorage.start(); storageManager.setRoot(rootNode); storageManager.storeRoot(); The node rootNode is passed to the StorageManager as root and saved. A subsequent loading of the root results in the previously created graph.\nAs we can see from this example, EclipseStore can store graphs without any problems. Cycles are acceptable here.\nConclusion: # We have now seen that EclipseStore can store any data structure in its entirety; lists, trees and graphs are not a limitation. This allows us to create the data model independently of the persistence layer. This is a possibility that I have always been looking for in the last 20 years of my Java activities.\nWe will now deal with the intricacies of EclipseStore in the following parts. It remains exciting.\nHappy coding\nSven\n","date":"6 October 2023","externalUrl":null,"permalink":"/posts/eclipsestore-storing-more-complex-data-structures/","section":"Posts","summary":"How to store complex data structures using EclipseStore? Are there any restrictions? How can you work more efficiently on such structures? We will now get to the bottom of these questions here.\n","title":"EclipseStore - Storing more complex data structures","type":"posts"},{"content":"","date":"6 October 2023","externalUrl":null,"permalink":"/tags/opensource/","section":"Tags","summary":"","title":"OpenSource","type":"tags"},{"content":"We will take the first steps with the Eclipse Store here. I will show what the Eclipse Store is, how you can integrate it into your project and what the first steps look like from a developer\u0026rsquo;s perspective. All in all, it is a practical introduction.\nWhat is Eclipse Store? # First, I would like to briefly explain what Eclipse Store actually is. Simply put, this is a mechanism for storing Java object trees. This initially sounds very theoretical, but it is much simpler than you might think.\nWhen I started developing Java applications in 1996, I dreamed of being able to easily store the objects I created in my application. I found the assumption that serialization could be used to write everything to the hard drive as a byte stream very good. Unfortunately, the reality looked different or still looks different today. You were quickly confronted with various technologies, all with their own characteristics. I still remember very clearly the first steps in which the application\u0026rsquo;s data had to be persisted via a JDBC interface. Who doesn\u0026rsquo;t know the JDBC-ODBC interface from back then? A typical application now had JDBC connections, mapping layers that implement everything in SQL and, of course, caches in various places so that the application\u0026rsquo;s responsiveness remained tolerable for the user. It\u0026rsquo;s not easy when dealing with a more complex data model.\nWith Eclipse Store, we now have a tool that eliminates most of these technology layers. So we have finally achieved what was promised to us in 1996.\nLet\u0026rsquo;s now come to the implementation of our own project. For this, we need the maven dependencies. For this example, I used the first final version. This is based on Microstream Version 8 and is the basis for the Eclipse project. As far as I know, Microstream is not being actively developed further.\n\u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.eclipse.store\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;storage-embedded\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;{maven-version}\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; Now that we have added the necessary dependencies to the project, we can start with the first lines of source code.\nBasic first Steps: # The Storage Manager: # We will now look at the essential elements of architecture. The linchpin of the persistence solution is the \u0026ldquo;Storage Manager\u0026rdquo;. This is the interface through which the behaviour can be configured. All entities are also transferred to persistence, retrieved and deleted. The storage manager itself is available in various forms. We will focus on embedded storage here for now. This implementation is an in-memory solution that accesses the local hard drive directly. You shouldn\u0026rsquo;t let the name confuse you. As is usual with RDBMS systems, it is not a tiny solution that allows you to do a little testing. The implementation of embedded storage is absolutely suitable for production.\nSo, an instance of this implementation is needed. The easiest way to get this is to call:\n**final****EmbeddedStorageManager storageManager = EmbeddedStorage.start();** The chosen implementation depends on which dependencies have been defined in the pom.xml. In our case, it\u0026rsquo;s the embedded implementation. We will discuss various other implementations in the following parts. We will also take a closer look at how the behaviour can be configured later. These parameters are irrelevant for the first steps and can safely be ignored.\nStorage Root: # The storage manager now needs to know the root of the graphs to be stored. You can imagine this as described below.\nIf we only have one element that needs to be stored, then no further wrapper is necessary. However, if we have two parts that we want to save, we need a container to hold both elements. This container is then the root through which initial access to all elements can occur. This can be a list or a class specifically tailored to your needs. This container must now be made available for persistence.\nLet\u0026rsquo;s look at an example. Two different strings should be stored here. A list is used as a container. This means that the instance of the list is the StorageRoot element. This must now be declared as an anchor point.\nEmbeddedStorageManager storageManager = EmbeddedStorage.start(); storageManager.setRoot(new ArrayList\u0026lt;\u0026gt;()); storageManager.storeRoot(); The “storeRoot()” instruction causes this modification to be saved at the root. We will see this “storeRoot()” method call more often. This persists in all changes in the graph, starting from the root. In our case, the root itself is saved.\nHow can another element be added to this container? There is no need to keep a reference to this container. Instead, we can ask the Storage Manager for this reference. What should not be forgotten at this point? If you overwrite the root, all previously saved elements will be deleted. These are then no longer under the supervision of persistence.\n**List \u0026lt;String\u0026gt; root = (List\u0026lt;String\u0026gt;) storageManager.root();** A downer at this point. Unfortunately, the \u0026ldquo;root()\u0026rdquo; method is not typed. Here, you have to know exactly what type the container is. We\u0026rsquo;ll look at how to deal with this better. I ask for patience here.\nAny number of elements can now be added or removed from this list. To save the changes, the StorageManager is informed of this. Let\u0026rsquo;s look at this in a little more detail. We will expand and modify the previous source code for this. Instead of simple strings, a separate class with two attributes is now used. The first attribute is again a string and represents the data value. The second attribute is an instance of the type LocaDateTime and is an automatically set timestamp. We can already see here that it is no longer a simple attribute. The constructor takes over the data value, which can be modified later. The timestamp cannot be set explicitly. There is only one getter here. Such constructs can also be processed by EclipseStore without any further action.\npublic class DataElement { private String value; private LocalDateTime timestamp; public DataElement(String value) { this.value = value; this.timestamp = LocalDateTime.now(); } public void setValue(String value) { this.value = value; this.timestamp = LocalDateTime.now(); } public String getValue() { return value; } public LocalDateTime getTimestamp() { return timestamp; } @Override public String toString() { return \u0026#34;DataElement{\u0026#34; + \u0026#34;value=\u0026#39;\u0026#34; + value + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, timestamp=\u0026#34; + timestamp + \u0026#39;}\u0026#39;; } Case I - We add elements to the container: # In our case, the simplest case is to add one or more elements to the root element, a list of type DataElement. This is done by adding the desired elements to the list and calling the “storeRoot()” method at the root level or saving the elements yourself.\nEmbeddedStorageManager storageManager = EmbeddedStorage.start(); storageManager.setRoot(new ArrayList\u0026lt;DataElement\u0026gt;()); storageManager.storeRoot(); List\u0026lt;DataElement\u0026gt; root = (List\u0026lt;DataElement\u0026gt;) storageManager.root(); root.add(new DataElement(\u0026#34;Nr 1\u0026#34;)); root.add(new DataElement(\u0026#34;Nr 2\u0026#34;)); storageManager.storeRoot(); And here is the method in which the elements are specified directly.\nDataElement d3 = new DataElement(\u0026#34;Nr 3\u0026#34;); DataElement d4 = new DataElement(\u0026#34;Nr 4\u0026#34;); root.add(d3); root.add(d4); storageManager.storeAll(d3, d4); Case II - We remove elements from the container: # The opposite operation of adding elements is deleting them. Here, too, the process is very straightforward. However, looking at the different options is unusual at the beginning. First, it\u0026rsquo;s the most intuitive version. Here, the element is removed from the holding object, the root itself. This reduced instance is then passed on to the StorageManager with the request to save this modification.\nroot.remove(d3); storageManager.store(root); However, you can also use the removed element yourself. For this purpose, the element previously removed from the holding instance is passed to the StorageManager for storage. This approach works but is anything but intuitive. At this point, for reasons of comprehensibility, you should not write it. Your colleagues will be grateful for that.\nroot.remove(d4); storageManager.store(d4); Case III - We modify the elements within the container: # Since elements can now be added and removed, the only thing missing is the modification. Here, too, the procedure is analogous to adding. The instance is modified and then passed to the StorageManager for storage. The saving process can, of course, also take place via a holding element. For readability reasons, however, saving the modified elements yourself is strongly advisable. Exception can, of course, be when you change a lot of elements within a holding instance.\nd3.setValue(\u0026#34;Nr 3. modified\u0026#34;); storageManager.store(d3); Fazit: # We have now seen how easy it is to get started with EclipseStore. Elements can now be saved without any further effort. This way, you can realize the first small applications. But that\u0026rsquo;s not the end of it. The following parts will address the intricacies and challenges of more complex applications. So it remains exciting.\nHappy Coding\nSven\n","date":"4 October 2023","externalUrl":null,"permalink":"/posts/how-to-start-with-eclipsestore-01/","section":"Posts","summary":"We will take the first steps with the Eclipse Store here. I will show what the Eclipse Store is, how you can integrate it into your project and what the first steps look like from a developer’s perspective. All in all, it is a practical introduction.\n","title":"How to start with EclipseStore - 01","type":"posts"},{"content":"Test-driven development (TDD) is a software development approach that prioritizes writing automated tests while creating the actual code. There follows a cycle of writing a failed test, writing the code to make the test pass, and then refactoring the code. TDD was originally developed to ensure the quality, maintainability and expandability of the software created over the long term. The specific knowledge about the individual source text passages should also be shown in the tests. Thus, a transfer of responsibility between developers is supported. Better than any documentation, tests are always up-to-date regarding the function that has been implemented in the source code.\nHowever, TDD also has a positive impact on the security of a program. And that\u0026rsquo;s what we\u0026rsquo;re going to look at now.\nBut first, let\u0026rsquo;s get back to TDD. There are a variety of theoretical approaches here. I will briefly touch on them here to give an overview. The question to be addressed is whether the TDD approach influences the effect.\nhttps://youtu.be/S-GKA6KJwPc\nClassic TDD - Kent Beck # Unit tests and units tested with them are continuously developed in parallel. The actual programming is done in small, repeating micro-iterations. One such iteration, which should only take a few minutes, consists of three main parts, known as the red-green refactoring.\nRed: Write a test to test a new behaviour (functionality) to be programmed. You start with the simplest example. If the feature is older, it could also be a known bug or new functionality to be implemented. This test is initially not fulfilled by the existing program code, so it must fail.\nGreen: Change the program code with as little effort as possible and add to it until it passes all tests after the test run then initiated.\nThen clean up the code (refactoring): remove repetitions (code duplication), abstract if necessary, align with the binding code conventions, etc. In this phase, no new behaviour may be introduced that would not already be covered by tests. After each change, the tests are performed; if they fail, it is impossible to replicate what appears to be a bug in the code being used. Cleanup aims to make the code simple and easy to understand.\nThese three steps are repeated until the known bugs are fixed, the code provides the desired functionality, and the developer can\u0026rsquo;t think of any more useful tests that could fail. The program-technical unit (unit) treated in this way is considered complete for now. However, the tests created together with her are retained to be able to test again after future changes as to whether the behavioural aspects that have already been achieved are still fulfilled.\nFor the changes in step 2 - also called transformations - to lead to the goal, each change must lead to a more general solution; for example, she may not only edit the current test case at the expense of others. Tests that get more and more detailed bring the code to a more and more general solution. Regular attention to transformation priorities leads to more efficient algorithms.\nConsistently following this approach is an evolutionary design method, where every change evolves the system.\nOutside in TDD # Outside-In Test-Driven Development (TDD) is an approach to software development that places emphasis on starting the development process first by creating high-level acceptance tests or end-to-end tests that demonstrate the desired behaviour of the system from his point of view to define users or external interfaces. It is also commonly referred to as behaviour-directed development (BDD).\nWith Outside-In TDD, the development process begins with writing a failed acceptance test that describes the desired behaviour of the system. This test is usually written from a user\u0026rsquo;s perspective or a high-level component that interacts with the system. The test is expected to initially fail as the system does not have the required functionality.\nOnce the first acceptance test has been performed, the next step is to write a failing unit test for the smallest possible unit of code that will pass the acceptance test. This unit test defines the desired behaviour of a specific module or component within the system. The unit test fails because the corresponding code still needs to be implemented.\nThe next step is implementing the code to make the failed unit test succeed. The code is written incrementally, focusing on the immediate needs of the failed test. The developer writes further tests and code step by step until the acceptance test is passed and the desired behaviour of the system is achieved.\nThe idea behind Outside-In TDD is to drive the development process from the outside, starting with the higher-level behaviour and moving inward to the lower-level implementation details. This approach helps ensure that the system is developed to meet the needs and expectations of its users. It also encourages component testability and decoupling by encouraging the creation of small, focused tests and modular code.\nBy practising outside-in TDD, developers can gain confidence in their code and ensure the system behaves as expected. It also helps uncover design flaws early in the development process and encourages the creation of loosely coupled and easily maintainable code.\nOverall, Outside-In TDD is a methodology that combines testing and development, focusing on providing value to users and driving the development process from an outside perspective.\nAcceptance Driven TDD # In test-driven development, system tests are continuously developed or at least specified before the system. In test-driven development, system development is no longer to meet written requirements but to pass specified system tests.\nAcceptance Test-Driven Development (ATDD), while related to Test-Driven Development, differs in approach from Test-Driven Development. Acceptance test-driven development is a communication tool between the customer or user, the developer and the tester to ensure the requirements are well described. Acceptance test-driven development does not require automation of test cases, although it would be useful for regression testing. The acceptance tests for test-driven development must also be legible for non-developers. In many cases, the tests of test-driven development can be derived from the tests of acceptance test-driven development. For this approach to be used for system security, the boundary conditions must go beyond the normal technical aspects, which are only considered in some cases.\nWhat test coverage should I use? # It is important for the effect on security that tests run automatically. These then lead to test coverage that can be measured and compared to previous runs. The question arises as to which test coverage will be most suitable. It has been shown that there are significant differences in the strength of the respective approaches. I am a strong proponent of mutation test coverage, as it is one of the most effective test coverages. If you want to know more here, I refer to my video on \u0026ldquo;Mutation Testing in German\u0026rdquo; or \u0026ldquo;Mutation Testing in English\u0026rdquo; on my YouTube channel. In any case, it is important that the better the test coverage, the greater the effect that can be achieved.\nHow exactly is the security of the application supported? # 1. Identify vulnerabilities early: Writing tests before implementing features help identify potential security gaps early. By considering security concerns during the test design phase, developers can anticipate possible attack vectors and design their code with security guidelines in mind. This proactive approach makes it possible to detect security problems before they become deeply embedded in the code base.\n2. Encouraging Safe Coding Practices: TDD encourages writing modular, well-structured, and testable code. This enables developers to adopt secure coding practices such as input validation, proper error handling, and secure data storage. By writing tests that simulate different security scenarios, developers are more likely to consider edge cases and verify that their code behaves securely.\n3. Regression Testing for Security: TDD relies on running automated tests regularly to maintain existing functionality as new code is added. This approach helps prevent a decline in security-related issues. If a security-related test case exists and initially passes, subsequent changes to it can be tested to ensure that vulnerabilities were not accidentally introduced.\n4. Building a Security-Aware Culture: TDD\u0026rsquo;s incorporation of security testing as an integral part of the development process helps foster a security-aware culture within the development team. By performing security-focused testing alongside functional testing, developers are more likely to prioritize security and see it as a fundamental aspect of their job. This mindset encourages a proactive attitude towards security rather than just seeing it as a side issue.\n5. Facilitate Collaboration and Code Review: TDD encourages collaboration and code review between team members. When developers first write tests, they can discuss security concerns and get feedback from peers. This collaborative approach increases overall security awareness within the team and enables early detection and resolution of security issues.\nAlthough TDD is not a security panacea, it offers a systematic and disciplined development approach that indirectly strengthens software security posture. By promoting security-centric thinking, promoting secure coding practices, and facilitating early detection of vulnerabilities, TDD can help build more secure and resilient applications. However, it is important to note that additional security measures such as threat modelling, code analysis, and penetration testing should also be included to meet security requirements fully.\nA few more practical approaches # What are Compliance Issues, and how to remove them? # The compliance issues are elements that run under a license and may not be used in this way in the project. This can happen in all technology areas within a project. Libraries or frameworks used do not usually change their license over the years, but it occasionally happens. All dependencies of the elements used must also be checked. And that\u0026rsquo;s up to the last link.\nOne must replace the affected element with a semantic equivalent available under an appropriate license to fix such a violation.\nWhat are Vulnerabilities, and how to remove them? # Vulnerabilities are unintentional errors in the source code that allow the system to be manipulated to its advantage. These errors can occur throughout the tech stack. It is important to recognize all vulnerabilities in all technologies used to get an overview of which vulnerabilities can be exploited in which combination for an attack on this system. Individual considerations need to be clarified at this point. This holistic view enables the recognition of the different attack vectors and the identification of weak points that cannot be exploited in this system. This allows you to use your efforts in a targeted manner to secure the system.\nWhy do we need efficient dependency management? # What tools are available to developers to address compliance issues and vulnerabilities? The answer is obvious. This is where very effective dependency management combined with robust test coverage comes in handy. Because, in all cases, dependencies have to be exchanged. Even though it\u0026rsquo;s a surrogate for compliance issues, fixing vulnerabilities is the same dependency in a different version. This can be a higher or lower version. Overall, it is about the clever combination of the various elements. Strong test coverage facilitates the subsequent test to determine whether the functionality is still available.\nConclusion: # In summary, strong test coverage is beneficial when modifying and exchanging external elements as well as your own source code. Here you can rely on the results of the CI environment without having to add manual verifications and thus implement a fast and lean release process with as little time loss as possible.\nQuality and safety support each other.\nHappy coding\nSven\n","date":"28 June 2023","externalUrl":null,"permalink":"/posts/tdd-and-the-impact-on-security/","section":"Posts","summary":"Test-driven development (TDD) is a software development approach that prioritizes writing automated tests while creating the actual code. There follows a cycle of writing a failed test, writing the code to make the test pass, and then refactoring the code. TDD was originally developed to ensure the quality, maintainability and expandability of the software created over the long term. The specific knowledge about the individual source text passages should also be shown in the tests. Thus, a transfer of responsibility between developers is supported. Better than any documentation, tests are always up-to-date regarding the function that has been implemented in the source code.\n","title":"TDD and the impact on security","type":"posts"},{"content":" Workshops # I am also happy to offer my knowledge in the form of workshops. I have compiled a list of topics for this. Workshops can take place on-site in your rooms or also virtually.\nIf necessary, the workshops can, of course, be individually adapted. Contact me so that we can coordinate the content together.\njUnit5 - Deep Dive\nUnit tests are a fundamental building block of the agile development process and a crucial factor for the quality and stability of your software. They make it possible to detect errors early and increase the software\u0026rsquo;s maintainability, which can reduce development costs in the long term. JUnit is the most widely used framework for testing Java applications, giving you all the tools you need to develop high-quality tests. With JUnit5\u0026hellip;\u0026hellip;\n— read more\nScreenshot\nread more\nStreams - Deep Dive\nThe field of application of streams in Java is enormous and can not only be used in new projects. Even old Java programs benefit from the streams. Here one speaks of refactoring, the modernization of existing source texts, some of which have been used for more than 10 years. With Java 8, streams and lambdas were included in the JDK. These two elements make it possible to write code that describes what should happen and not how it should happen. This makes it easier to formulate declarative source text in Java\u0026hellip;\u0026hellip;\n— read more\nread more\nFunctional Programming in Core Java\nJava has always been an object-oriented programming language. What was novel at the time and taken as a big argument for Java has been used as a counter-argument from time to time in the recent past. Again and again, the advantages of functional programming come up for discussion. This is accompanied by whether Java should still be the programming language for the new project? What about old projects? Do you have to do without functional programming because Java can\u0026rsquo;t just be replaced?\n— read more\nread more\n","date":"14 June 2023","externalUrl":null,"permalink":"/workshops/","section":"Workshops","summary":"Workshops # I am also happy to offer my knowledge in the form of workshops. I have compiled a list of topics for this. Workshops can take place on-site in your rooms or also virtually.\nIf necessary, the workshops can, of course, be individually adapted. Contact me so that we can coordinate the content together.\n","title":"Workshops","type":"workshops"},{"content":" Description:\n\u0026ldquo;Java Streams - Deep Dive\u0026rdquo;\nThe field of application of streams in Java is enormous and can not only be used in new projects. Even old Java programs benefit from the streams. Here one speaks of refactoring, the modernization of existing source texts, some of which have been used for more than 10 years.\nWith Java 8, streams and lambdas were included in the JDK. These two elements make it possible to write code that describes what should happen and not how it should happen. This makes it easier to formulate declarative source text in Java.\nThe big goal is to produce more understandable and maintainable source code.\nIn this workshop, we will deal with the streams in detail. How they work, why they are not a data structure, and what a big difference they make to the existing data structures. What does the API look like, and which \u0026ldquo;best practices\u0026rdquo; have emerged in recent years.\nHere are just a few keywords of the topics that will be covered.\nLambdas, in their various forms Difference to anonymous inner classes Handling of primitive data types Method references Fundamentals of functional programming Optionals and the avoidance of null pointer exceptions serial and parallel streams and some more Who is this workshop suitable for?\nThe workshop is for anyone who wants to deal with streams more intensively. It is equally suitable for new projects and challenges arising from the maintenance of old existing systems.\nAfter this workshop, there is a good understanding of streams, their areas of application, possibilities, but also their limitations.\nThrough the practical exercises, the necessary experience is gained to start using the streams immediately in your project.\nLanguage :\nGerman or English\nWorkshop Duration:\n1 day\nLocation\nIn house or remote\n","date":"14 June 2023","externalUrl":null,"permalink":"/workshop-java-streams-deep-dive/","section":"Workshop - Java Streams - Deep Dive","summary":" Description:\n“Java Streams - Deep Dive”\nThe field of application of streams in Java is enormous and can not only be used in new projects. Even old Java programs benefit from the streams. Here one speaks of refactoring, the modernization of existing source texts, some of which have been used for more than 10 years.\nWith Java 8, streams and lambdas were included in the JDK. These two elements make it possible to write code that describes what should happen and not how it should happen. This makes it easier to formulate declarative source text in Java.\n","title":"Workshop - Java Streams - Deep Dive","type":"workshop-java-streams-deep-dive"},{"content":" Description:\n\u0026ldquo;Functional Programming in Core Java\u0026rdquo;\nJava has always been an object-oriented programming language. What was novel at the time and taken as a big argument for Java has been used as a counter-argument from time to time in the recent past. Again and again, the advantages of functional programming come up for discussion. This is accompanied by whether Java should still be the programming language for the new project? What about old projects? Do you have to do without functional programming because Java can\u0026rsquo;t just be replaced?\nThe good thing is that since Java8, some syntactical elements have made their way into the language, which allows you to get very close to functional programming. Many of the syntactic advantages are now implementable.\nHow does that work? Do I need a new library for this, an extensive framework? My answer to that is clearly no.\nIn this workshop, we will only work with the JDK8 or higher and learn the functional aspects step by step. In addition to the theory, we will go through many practical exercises that will enable you to implement the newly discovered elements yourself and thus consolidate them.\nTopics that we will examine in more detail\nHandling exceptions Dealing with zero What is a function Higher order function Memoizing Currying Connection to the object-oriented source text Connection to streams Transformation of decision trees Who is this workshop suitable for?\nThe workshop is aimed at anyone who would like to deal more intensively with the functional aspects of Java. It is equally suitable for new projects and challenges arising from maintaining old legacy systems.\nAfter this workshop, the foundations have been laid so that you can start directly implementing the learned approaches in your project.\nLanguage :\nGerman or English\nWorkshop Duration:\n1 day\nLocation\nIn house or remote\n","date":"14 June 2023","externalUrl":null,"permalink":"/workshop-functional-programming-in-core-java/","section":"Workshop - Functional Programming in Core Java","summary":" Description:\n“Functional Programming in Core Java”\nJava has always been an object-oriented programming language. What was novel at the time and taken as a big argument for Java has been used as a counter-argument from time to time in the recent past. Again and again, the advantages of functional programming come up for discussion. This is accompanied by whether Java should still be the programming language for the new project? What about old projects? Do you have to do without functional programming because Java can’t just be replaced?\n","title":"Workshop - Functional Programming in Core Java","type":"workshop-functional-programming-in-core-java"},{"content":"","date":"10 December 2022","externalUrl":null,"permalink":"/categories/devsecops/","section":"Categories","summary":"","title":"DevSecOps","type":"categories"},{"content":"Supply Chain Security is a hot topic these days. And more and more, we as developers are dealing with this daily. But what does this mean for us, and how is this influencing our job? I want to give an overview of common attacks against the Software Supply Chain from the developer\u0026rsquo;s view and will introduce the Open Source project SLSA from the Linux Foundation.\na) Who is the project SLSA\nVarious experts from the security field started this project to share their knowledge, leading to the mentioned project. There is no company or governmental organisation behind this project. It is a pure Open Source project under the umbrella of the Linux Foundation.\nb) What is the goal of this project?\nThe SLSA project is open-source and will not provide its own source code. It is, therefore, not a classic open-source project to publish a specific solution. Instead, it is a documentation project to process knowledge about supply chain security in software development and make it freely accessible. The structure contains examples of where the events or attacks mentioned have been successfully taken and which countermeasures should be taken. We\u0026rsquo;ll look at the security levels provided a little later. The aim here is to allow the reader to prepare for these threats with specific steps slowly.\nAfter all, based on your options and the current environment, you have to decide which next steps and measures make sense to implement.\nhttps://youtu.be/r8moPP8EACc\nc) What is the current status of the project?\nThe project is currently (beginning of 2022) still in the alpha phase. There is a lot of documentation, but some places still provide references to future content. So it\u0026rsquo;s always worth taking a look inside. Since this is an open-source project, you can, of course, also contribute yourself. Here you have the opportunity to make your expertise available to others.\nSLSA security levels?\na) Why do we need these security levels?\nWhen you look around the SLSA project website for the first time, the description of the security levels quickly catches your eye. But what are these levels all about? With these levels, you want to achieve two things. First, it should help to enable a self-assessment. This first consideration serves to classify the own existing environment and to identify the primary weak points. The second goal of this level is to help you plan the following purposes that should be implemented to increase your protection significantly. These recommendations are based on evaluating many projects and environments and on the security experts\u0026rsquo; knowledge. This enables you to use the available means and opportunities to increase your security.\nb) Level 0\nThe first level mentioned is marked with the number zero. This is the state in which no action has yet been taken. The documentation that describes the entire build process must be available and up-to-date to reach this level. This enables an initial analysis of the environment to be secured. It can also be seen as a way of taking stock.\nc) Level 1\nThe level with the number one requires first actions on the environment. The point here is to list all the components used to create a binary file. Not only are the dependencies used listed, but all other metadata is recorded if possible. There is the term SBOM, Software Bills Of Materials in the industry. However, the term can be broadened considerably here. Within JFrog Artifactory, there is \u0026ldquo;build info\u0026rdquo;, which means a superset of SBOM. All sorts of metadata are recorded here, such as the operating system and version of the build machine, environment variables and their contents, etc. I have created my video for this.\nIn this video, I\u0026rsquo;ll go over the details of SBOM and Build Info.\nhttps://www.youtube.com/watch?v=ILgZexU07dc\nWithin the SLSA project, the term SBOM or Build Info is not explicitly mentioned. Instead, the requirement is described more generally, and it is also clearly pointed out that this level does not yet help against the compromise itself. It is only the basis for the follow-up and the comprehensive risk assessment. Based on this level, one can start with vulnerability management.\nd) Level 2\nLevel two describes code versioning software in combination with a hosted build service. With this combination, you want to trust the build service and thus trust the origin of the generated files. Here you must take a critical look at where and who will offer these services. The fundamental consideration is based on the assumption that a specialized provider has a better grip on the issue of security within the build infrastructure than you can guarantee yourself. However, the manipulation of protection of the build service can be increased in various ways.\ne) Level 3\nWith level three, auditors are now included in the considerations. Here, trained security specialists are used to check the existing build environment for possible vulnerabilities. As part of an accreditation process, the auditors determine whether specific standards are met to ensure the verifiability of the source and the integrity of the provenance (origin). This level offers better protection than all previous levels to ward off well-known threats. Preventing cross-build contamination is an example.\nf) Level 4\nIn the SLSA project, level four is the highest currently achieved. To reach this level, the following conditions are set. The first requirement describes the necessity that two people constantly carry out all changes. This does not only refer to the changes made to the source code. The term is broader here and also includes all changes to the system and the configurations. This ensures that no intentional or unintentional compromise can occur at any point. Such a process has prevented most bugs, vulnerabilities and compromise attempts.\nThe second requirement relates to the technical environment. It is assumed here that it is a hermetic and reproducible creation process. However, a distinction should be made between the requirement for hermetic builds and reproducible builds. The former guarantees that all required dependencies are known and checked for vulnerabilities. Here, of course, it plays a role from which sources these dependencies were obtained. On the other hand, reproducible builds help detect changes and conduct subsequent analyses. However, the reproducibility of a build is not a mandatory part of the fourth level.\ng) Limitations\nEven if many supply chain threads are addressed with the SLSA project, there are some limitations that I would like to list here.\n1) The list of all dependencies in a project can simply become too large to be able to fully record and evaluate them. That can depend on different things. The point that shows this most clearly is based on the possibility of some dependency management systems not only specifying explicit dependency versions. In some scenarios, you can also define a version range. This now results in an exponential increase in all possible version combinations.\n2) The number of all components to be backed up can exceed the available capacities. The teams then have to decide which elements are subjected to safety analysis.\n3) The SLSA security levels are not transitive. This means that this component itself consists of components that have a lower security level. So here, you have to pay very close attention to the details.\n4) Even if automation is the solution in many areas, there may be areas where this will not be possible. Again, this refers in many areas to closed-source projects. Here, some analyzes are simply not feasible. It must then be decided how this dependency will be classified on a case-by-case basis.\n**What are the requirements? **\nAll the requirements mentioned here are a little more detailed in reality. An overview of the requirements was provided to guide what exactly is meant by each point. Here, you can see which condition belongs to which level and how this is understood. Then the details are discussed, such as what difference it makes if you only use a building service or if you also use the \u0026ldquo;Build as Code\u0026rdquo; strategy. All the details are too extensive here, so I refer to the section on the project\u0026rsquo;s website.\nSupply Chain Threads\n**a) Overview **\nTo better understand the project and its limitations, you should look at the individual types of attacks on a typical software development environment. A distinction is generally made between source, image and dependency threads. The SLSA project also delimits precisely what it includes and what it does not.\nThe following overview comes directly from the project and will be further developed if necessary. At this point, one must not forget that this project is only in the alpha phase (early 2022). However, these types of attacks mentioned are timeless and apply to almost every development environment.\nLet\u0026rsquo;s get to the source threads. This type of attack aims to compromise the source code itself.\nb.1) Bypass Code Review\nThe attack called \u0026ldquo;Bypass Code Review\u0026rdquo; describes a scenario in which an attempt is made to bypass the control mechanisms and directly add compromised source code to the project. Here public paths are taken to reach the destination. Such attacks were carried out on the Linux kernel, for example, via the mailing list. The best and probably simplest way to thwart such attacks is to have all source code changes checked independently by at least two people. However, at this point, one must also find ways to prevent mutual \u0026ldquo;covering up\u0026rdquo;.\nb.2) Compromised Source Control System\nIf importing compromised source code does not work, you can try to attack the CVS itself. Attempts are being made here to make changes to the source code on the server, bypassing the official channels. A prominent example is a successful attack on the source code of PHP. The self-managed system was attacked, and the changes were imported directly as a commit. Only securing the system helps against this type of attack. To manage such a system yourself, you must have the necessary resources and knowledge. In this case, it is often easier and cheaper to use a managed system with Github.\nNext, we come to the build threads.\nb.3) Modified Source Code after Source Control\nThe source code cannot only be changed where it is stored. When a Build process is started, the source code is fetched from the specified source. The source text on the build page can also be modified before the translation process starts. One attack in which this approach was used is the \u0026ldquo;Webmin\u0026rdquo; project. Here, additional and modified source texts were injected into the translation process. The agents responsible for carrying out the individual build steps must be secured to prevent this attack. Here, too, one has to weigh up how much of this work is carried out by oneself since how safe such an environment depends primarily on one\u0026rsquo;s own abilities.\nb.4) Compromised Build Platform\nSince the built environment is part of the value chain, this part must also be secured classically. Here, too, it is again the case that one\u0026rsquo;s own abilities in ​​safety must be assessed. A well-known representative of this type of attack is the Solarwinds Hack. Here, the build route was modified to compromise the newly created binary with each run.\nb.5) ​​Using Bad Dependencies - Dependency Thread\nAt this point, I would like to insert the dependency threads briefly. Dependencies are specifically modified and then offered. The security of the repositories plays a central role here. I\u0026rsquo;ll come back to that in a moment. However, this attack pattern refers to external locations where dependencies are fetched to process them. Particular attention should be paid to assessing the integrity of caches and mirrors of known official bodies. Often the \u0026ldquo;official\u0026rdquo; mirror servers of publicly accessible repositories do not have the same financial resources as the original. This causes attacks to be placed there.\nYou may be offered modified versions of known commonly used dependencies when you dock at such a place.\nb.6) Bypassed CI/CD\nLet\u0026rsquo;s get back to the build threads. Sometimes it is easier to bypass the CI/CD environment at an appropriate point. Here, the topics within a development path are selected where a transfer between the build server and the associated repositories is not sufficiently secured. The attack involves offering a recipient such as a repository a modified binary to make it look like it is from the CI environment. Malicious code is injected parallel to its own build infrastructure.\nb.7) Compromised package repo\nA component repository is also part of the infrastructure and can fall victim to an attack. As with all other elements, protection is necessary here. Attackers try to gain access to the repositories to replace the binaries of known and frequently used dependencies with modified versions. Depending on your ability to harden such a system against attacks, you have to decide whether an externally managed alternative will increase the security standard of your value chain.\nb.8) Using a bad package\nWe come to the last attack variant mentioned here. In this scenario, known dependencies are taken as a template and then their own modified versions are offered in a public repository. Here, an attempt is made to give the own variant a name close to the original name and depicts typical spelling errors. This dependency can still be resolved if a misspelling of the original name is used in a version definition. Only in this case a modified version of it will be loaded.\nThe only way to protect yourself against this attack is to carefully check how each individual dependency has been defined in your own projects.\nProject Persia Overview\nWe have seen which general types of attacks on a classic software development environment are possible. These are always generic attack patterns that can be used independently of the current products. Others also saw this as an idea to develop a general strategy to counter this. The language here is from the open-source project Persia, which is currently (early 2022) in the alpha phase. We are close to officially accepting it as an incubator in the CD Foundation. The project website is https://pyrsia.io.\nThe basic idea of ​​this project is that there will be a network of trust for open-source projects. A P2P network consisting of build and distribution nodes should exist where the sources are fetched, built and offered decentrally. In addition to the classic advantages such as reliability and better utilisation of the network resources of a P2P network, increased security should also be achieved with the binaries themselves. The types of attacks on source code management software are excluded. But all subsequent attack patterns can be repelled in this way.\nUnfortunately, it is not possible to fully describe the project. But at this point, I would like to draw attention to my video about this project. (https://pyrsia.io/)\nhttps://youtu.be/M3QQww1JzlU\nCheers Sven\n","date":"10 December 2022","externalUrl":null,"permalink":"/posts/introduction-to-the-linux-foundations-slsa-project/","section":"Posts","summary":"Supply Chain Security is a hot topic these days. And more and more, we as developers are dealing with this daily. But what does this mean for us, and how is this influencing our job? I want to give an overview of common attacks against the Software Supply Chain from the developer’s view and will introduce the Open Source project SLSA from the Linux Foundation.\n","title":"Introduction to the Linux Foundation's SLSA project","type":"posts"},{"content":"Intro\nThis article will take a detailed look at what the term build-info is all about and why it will help us protect against attacks such as the Solarwinds Hack.\nWhat is the concept behind the term - build-info?\nLet\u0026rsquo;s start at the very beginning and clarify the basic principle behind the term build-info. The term build-info has been coined for many years by the company JFrog, among others. This is a particular type of repository.\nThis repository stores the information that describes the context that led to the creation of a binary file. With this information, you can now achieve a wide variety of things.\nhttps://youtu.be/ILgZexU07dc\nWhat components does build-info consist of?\nThe content of a build-info is not strictly defined. Instead, the approach that applies is that the more, the better. Of course, you have to proceed with caution here too. All possible parameters are collected. In addition to the date and time, the system on which the process was run, which operating system was used in which patch level, to active environment variables, compiler switches and library versions.\nThe challenge is actually that it is not known which information will later be helpful and expedient. For this reason, more rather than less should be saved.\nWhy do we actually need a build-info?\nThe task of a build-info is to enable the observation, or rather, the analysis of a past situation. There can be a variety of reasons for this. For example, it can be used to improve quality, or it can be the basis for reconstructing a cyber attack that has taken place. And with that, we come straight to the event that got everything rolling in the recent past.\nTrigger - SolarWinds Hack\nOne of the others will have heard or read something about it. We are talking about one of the most significant cyberattacks that have ever taken place. It\u0026rsquo;s the SolarWinds Hack. Here it was not the final target that was attacked directly, but a point in the supply chain. SolarWinds is a software company that provides a product for managing network infrastructure. With just over 300,000 customers worldwide, this software\u0026rsquo;s automatic update process has been the target of the attack. It was not the update process itself that was compromised, but the creation of the binaries that will be distributed with this update process. The attack took place on the company\u0026rsquo;s CI route to immediately infect the newly created binaries with each build. Here the CI route was manipulated so that another component was added to the binary to be generated. This component can be thought of as a kind of initial charge. As soon as this has been ignited or activated, further components are dynamically reloaded. As a result, each infection had different forms. These files were then offered to all customers by means of an automatic update. Thus, over 15,000 systems were infiltrated within a short time.\nReaction - Executive Order of Cybersecurity\nSince there were many well-known US companies, US organizations and US government institutions among the victims, the question arose of how to counter such a threat from the US state in the future. The US government has decided that one begins with the complete cataloguing of all software components in use, including all their constituent parts. This obligation to provide evidence was formulated in the \u0026ldquo;Executive Order of Cybersecurity\u0026rdquo;. However, when I first heard about an executive order, I wasn\u0026rsquo;t sure what that actually meant.\nWhat is an executive order?\nAn executive order is a decree of the US President that regulates or changes internal affairs within the state apparatus. You can think of it as a US president like a managing director of a company who can influence his company\u0026rsquo;s internal processes and procedures. In doing so, no applicable law can be circumvented or changed. With such an executive order, no law can be changed, stopped or restricted. But it can change the internal processes of the US authorities very drastically. And that\u0026rsquo;s exactly what happened with this Executive Order of Cybersecurity, which has directly impacted the US economy. Every company that works directly or indirectly for the state must meet these requirements to continue doing business with the US authorities.\nWhat is the key message of the Executive Order?\nThe Executive Order of Cybersecurity contains a little more text, the content of which I would like to shorten here and reproduce without guaranteeing legal correctness.\nThis arrangement aims to record the software operated by or for the US authorities in its entirety. This means that all components of the software used must be documented. This can be thought of as follows; If you want to bake a cake, you need a list of ingredients. However, listing these would not meet the requirements, as knowledge of all elements down to the last link is required. In our example, an ingredient that consists of more than one component would have to provide a list of all existing parts. This is how you get all the components used that are needed for the cake. If we now transfer this to the software, it is necessary to record all direct and indirect dependencies. The list of all components is then called - SBOM (Software Bills Of Material)\nAnd what does that mean for me?\nNow I am not directly working for an organ of the US government. But it will still reach me sooner or later. Because through the indirection, all possible economic sectors worldwide will be affected. It can also affect me in the private sector. If I have an open-source project that is used indirectly in this environment, it can only continue if I prepare the project accordingly. Long story short; It will come our way in whatever way it will happen.\nWhat are the general requirements for a build-info?\nLet\u0026rsquo;s get back to the build-info. A build-info is a superset of an SBOM (Software Bills Of Material). So the dependencies are catalogued, and additional information from the runtime environment in which the binary is generated is recorded. However, there are also some requirements for such build-info. By this, I mean properties that must be present in order to be able to deal with this information in a meaningful way.\nAccessibility:\nThe information that is collected must be quickly and easily accessible and therefore usable. A central storage location such as Artifactory is suitable here, as all binaries and dependencies are also stored here. You also have all the meta-information of a respective binary file at this location.\nImmutability:\nWhen information is collected, it makes sense to store it in such a way that it can no longer be changed. This promotes the acceptance of the data situation and gives particular security when evaluating a case.\nCombinability:\nThe static data are also suitable for enriching them with current secondary data. In this case, I am talking about the vulnerabilities that can be found in the dependencies used. In contrast to the static information, this image has to be constantly updated. In practical use, this means that the data must be stored in a machine-readable manner in order to enable the connection of other data sources. In the case of Artifactory, it is the data from JFrog Xray that is displayed here.\nHow can a build-info be generated?\nIt would be best if you had the following things.\nFirst, the JFrog CLI generates the build-info and second, it is possible to save this information permanently. You can do this very quickly with the Build-Info Repositories from Artifactory. But one step at a time.\nThe process for generating a build-info is as follows.\nIn the following example, we assume that we are going to build a Java project with maven. Here you have to get involved in the build process. You can do that quite easily with the JFrog CLI tools. These represent a wrapper for the build process used in each case. In our example, it is maven. To install the JFrog CLI Tools, you can go on the website https://jfrog.com/getcli/ choose the installer that is suitable for it. After installing and configuring the JFrog CLI Tools, you can start the build via the CLI on the command line. After a successful build, you can find the build-info as a JSON file in the target directory. This file can be viewed with a text editor to get a feel for the wealth of information that is stored here. This file is regenerated with each build. This means that there is a 1: 1 relationship between the build cycle and the binaries generated in the process. Using the CLI, you can then transfer this information to the Artifactory build repository and evaluate it there using a graphical user interface.\nHow can I link vulnerability scans to the build-info?\nIf you are now within Artifactory (https://bit.ly/SvenYouTube ), look at the build-information; you also get the current information regarding the known vulnerabilities. This is an example of how the build-information can be enriched with further system information.\nSince there is not enough space at this point to clarify the generation and evaluation, I refer to a demo project in which I have stored the steps in the README, including detailed explanations of the individual practical steps.\nIf you want, you can also register for one of my free DevSecOps workshops. You are also welcome to speak to me directly if you\u0026rsquo;re going to do this in a free community or company event.\nThe URL for the demo project is:\nhttps://github.com/Java-Workshops/JFrog-FreeTier-JVM\nHave fun trying it out.\nCheers, Sven!\n","date":"8 October 2021","externalUrl":null,"permalink":"/posts/the-power-of-jfrog-build-info-build-metadata/","section":"Posts","summary":"Intro\nThis article will take a detailed look at what the term build-info is all about and why it will help us protect against attacks such as the Solarwinds Hack.\n","title":"The Power of #JFrog Build Info  (Build Metadata)","type":"posts"},{"content":" In the past two years, we have had to learn a lot about cybersecurity. The new attack vectors are becoming more and more sophisticated and are directed more and more against the value chain in general. But what does that mean for us? What can be done about it, and what reactions have the state already taken?\nLet\u0026rsquo;s start with the story that got all of this rolling and made sure that the general attention was drawn to the vulnerabilities of the available IT infrastructure. We\u0026rsquo;re talking about the SolarWinds Hack. What happened here, and what is more critical; What are the lessons learned from this incident?\nhttps://youtu.be/ClYhATBlBl0\nIt is essential to know that the SolarWinds company produces software that is used to manage network infrastructure. With the name \u0026ldquo;Orion Platform, \u0026quot; this software should help manage and administer the network components efficiently. If you look at the product description on the SolarWinds website, you can read that the software can monitor, analyse, and manage the network infrastructure. And that\u0026rsquo;s exactly what the customers did. The company itself has around 300,000 customers worldwide, including individual companies and government organisations and corporations from a wide variety of areas. To always stay up to date, the software platform includes an automatic update mechanism. And that was exactly what the attackers were after. They saw in the SolarWinds company a multiplier for their own activities. But how can you go about this? The attackers obtained the necessary software tools by breaking into the FireEye company and infiltrating the SolarWinds network. The goal was the software development department in which the binaries of the Orion platform are created. Here, the CI routes have been manipulated to include compromised binaries in every build of the software. As a result, the company produced these binaries and put them into circulation through the automatic update process. In this way, around 18,000 targets could be infiltrated and compromised within a short time.\nWhat does that mean for us now?\nThere are two angles here. The first is from the consumer\u0026rsquo;s point of view. However, for your own production, it must be ensured that no compromised binaries are used. That means that all, and here I mean all binaries used, must be checked. These are the dependencies of the application and the tools used in the production process. This includes the tools such as the CI server used, e.g. Jenkins, or the operating system components of the infrastructure and Docker images.\nThe second perspective represents the point of view that one has as a software distributor. This means anyone who distributes binaries in any form. This includes customers in the classic sense and other departments that may be considered consumers of the binaries created. The associated loss of trust can cause a company severe financial damage.\nWhat can we do to protect ourselves?\nIf you look at the different approaches to arm yourself against these threats, you can see two fundamental differences.\nDAST - Dynamic Application Security Testing\nAn application can be subjected to a black box test. This is called DAST, Dynamic Application Security Testing. Here you go the way a hacker would do it. The application is attacked via the available interaction points. Attack vectors based on the most common vulnerabilities are usually used here. SQL injection can be cited as an example. These attack patterns are initially independent of the technology used and represent typical security holes in web applications.\nThis approach has advantages as well as disadvantages. A serious disadvantage is that the tests cannot be started until the application has been developed. This means that the measures to eliminate weak points found are more costly than is the case with other approaches. On the other hand, a significant advantage of this approach is that the dynamic context can also be checked when testing the production system. Examples include weak configurations, open ports, and more.\nSAST - Static Application Security Testing\nThe counterpart to the dynamic tests is the static tests. These tests analyse the individual components of the infrastructure involved. This includes not only the applications created but also the components involved in operating and creating the application.\nA disadvantage of the SAST method is that it can only search for known security vulnerabilities. However, this point is put into perspective in that the number of known security gaps increases steadily. Still, it is also assumed that the number of known security gaps exceeds the number of unknown security gaps. Another advantage of the SAST method is that it checks the licenses used. This area also contributes to the security of the company, albeit in a legal sense.\nSo what has the most significant effect when you start with the topic of security?\nWhen looking at the different approaches, it becomes clear that the proportion of directly or indirectly used binaries makes up the most significant amount in a technology stack. This means that you should also start securing precisely these parts. Unfortunately, there is only one way to ensure with SAST that it is really 100% checked directly. All other methods do not achieve this coverage, or if they do, then only in sporadic exceptional cases.\nHowever, it should be noted that one understands the interdependencies. This is the only way to identify and extract the possible attack vector. Here it is of essential importance that not only the binaries themselves are examined, but also the associated meta-information is evaluated. Within software development, there are two points for these analyses where such an analysis fits very well. The first point that most people will think of is the CI segment. Here the CI server can carry out the analysis during the execution of a build pipeline and all the checks and also take over the reporting. If the violations are too massive, the build is cancelled. The second position, which is ideally suited for such an analysis, is directly in the IDE. Here it can already be determined during the definition whether the license used is permitted. The known vulnerabilities can also be extracted and thus provide enough information to decide whether this version of this dependency may be used.\nLesson Learned - what we learned from the SolarWinds Hack.\nWe learned the following things from SolarWinds. First, the attacks are no longer aimed at the final targets themselves but against the suppliers. Here one tries to reach multipliers who massively increase the effectiveness of this attack. Likewise, you no longer only have to protect yourself against your use of compromised binaries. The newly added component means that the self-generated binaries must also be secured. Nobody wants to become a distributor of infected software. It follows that all parts involved must be subjected to permanent checks.\nReactions - The Executive Order from Mr Biden\nOne or the other has undoubtedly noticed that the US President gave this Executive Order of Cybersecurity. But what exactly does that mean, and who can even do something like that under what conditions?\nIn the United States, the president - the \u0026ldquo;executive branch\u0026rdquo; of the US government - can issue what is known as an executive order. Much like a CEO in a company, the executive branch has the power to make some unilateral decisions that affect the behaviour of the US government. These orders must be limited to government conduct and policies and not to general laws or statutes that encroach on the rights of any individual or state. If the executive transgresses in this area, the US Supreme Court or other legal remedies can appeal and revoke all measures. Only the president - due to his status as head of state - can issue an executive order. No other officer is allowed to do this.\nSignificantly, executive orders are mostly limited to how the US government operates internally. For example, an executive order might not simply resolve to raise taxes in a state or issue far-reaching commercial guidelines. However, executive orders are legally binding on the behaviour of the US government if they do not violate fundamental rights or make something unconstitutional. Because of this, these commands can be found to change the policy from the president to the president as, like a CEO, they set some of the government\u0026rsquo;s agendas. In this case, the Executive Ordinance refers to how the US government monitors and regulates its software usage and sets out requirements for how that software must be documented to be sold to the government. It doesn\u0026rsquo;t dictate trading on a broad scale.\nSince the US government can define its way of working, it was decided that the SBOM requirements (in preparation) will be the only way to approve software purchases - to get an idea of ​​the volume, you have to know it is one amount in the tens of billions per year. This means that any company that wants to sell to US government agencies, likely state and local agencies, and anyone who works with resellers who sell to the government and more, must meet all of these new standards.\nWhen the European Union agreed on comprehensive data protection and data security laws with the GDPR, the effect was felt worldwide and quickly. In a globally networked economy, decisions by the world\u0026rsquo;s largest GDP producers can set standards for other areas. With the GDPR, companies in the US and elsewhere had to immediately revise and update software and processes to continue doing business in the EU. It became a worldwide rule so as not to lose EU treaties. Likewise, the US cybersecurity order is believed to be reversing, with U.S.-based global corporations influencing global software security behaviour.\nWow - what now?\nWhat does that mean for me as a software developer? Well, the effects at this point are not as severe as you might want to imagine. In the end, it is said that a complete list of all contained dependencies must be created. This means the direct and indirect dependencies. To obtain such a dependency graph, one can, for example, use the tools that already do this. It means the tools that can generate a complete dependency graph, for example. For this, you need a logical point through which all dependencies are fetched. A tool such as Artifactory is best, as this is where the knowledge of all dependency managers used comes together. Devices such as the vulnerability scanner JFrog Xray can then access this information, scan for known vulnerabilities and licenses used, and extract the entire dependency graph. This information can thus be generated as build information by the CI route (e.g. JFrog Pipelines) and stored in a build repository within Artifactory.\nConclusion\nWe have seen that the new qualities of attacks on software systems, in general, will lead to advanced countermeasures. What is meant here is a complete analysis of all artefacts involved and produced. The tools required for this must combine the information into aggregated impact graphs across technology boundaries. One point within software development that is ideal for this is the logical point via which all binaries are fed into the system. However, the components alone are insufficient since all indirect elements can only be reached and checked by understanding the associated meta-information.\nThe tools used for this can then also be used to create the SBOMs.\nIn my opinion, it is only a matter of time before we also have to meet this requirement if we do not want to suffer economic disadvantages from the Executive Order of Cybersecurity.\nIf you want to find out more about the topic or test the tools briefly mentioned here, don\u0026rsquo;t hesitate to contact me.\nIn addition to further information, we at JFrog offer free workshops to prepare for these challenges in a purely practical manner.\nAnd if you want to know more immediately, you are welcome to go to my YouTube channel. There I have other videos on this topic.\nIt would be nice to leave a subscription on my YouTube channel as a small thank you. https://www.youtube.com/channel/UCNkQKejDX-pQM9-lKZRpEwA\nCheers Sven\n","date":"27 July 2021","externalUrl":null,"permalink":"/posts/solarwinds-hack-and-the-executive-order-from-mr-biden-and-now/","section":"Posts","summary":" In the past two years, we have had to learn a lot about cybersecurity. The new attack vectors are becoming more and more sophisticated and are directed more and more against the value chain in general. But what does that mean for us? What can be done about it, and what reactions have the state already taken?\n","title":"SolarWinds hack and the Executive Order from Mr Biden -- And now?","type":"posts"},{"content":"","date":"19 July 2021","externalUrl":null,"permalink":"/tags/dast/","section":"Tags","summary":"","title":"DAST","type":"tags"},{"content":"","date":"19 July 2021","externalUrl":null,"permalink":"/tags/iast/","section":"Tags","summary":"","title":"IAST","type":"tags"},{"content":"","date":"19 July 2021","externalUrl":null,"permalink":"/tags/rasp/","section":"Tags","summary":"","title":"RASP","type":"tags"},{"content":"","date":"19 July 2021","externalUrl":null,"permalink":"/tags/sast/","section":"Tags","summary":"","title":"SAST","type":"tags"},{"content":" Intro:\nIn this post, we\u0026rsquo;re going to look at the differences between the various cybersecurity defence techniques. Here you can identify four main groups, which we will go through briefly one after another to illustrate the advantages and disadvantages.\nhttps://youtu.be/sW7mTNVIUhE\nSAST - Static Application Security Testing\nSAST describes the process in which the components of an application are subjected to a static analysis. This approach not only searches for security gaps but also determines the licenses for the individual elements. In the following, however, I will only deal with the consideration of vulnerabilities in this post.\nThe term static comes from the fact that only the static context is used for this evaluation. All dynamic or context-related assessments are not possible with this method. SAST can be used to analyze the entire tech stack of an application if access to these components is possible, which is also one of the advantages of SAST compared to other analysis methods. The only exception is the IAST procedure, which we will come back to later.\nWhat exactly happens now with the SAST procedure? SAST is available in two forms. The first that I would like to mention relates to checking your source code. Here the attempt is made to determine whether there are patterns that make a later vulnerability easier or even possible in the first place. For example, it searches for calls to critical libraries or identifies unsafe handling of resources. However, one honestly has to admit at this point that, first of all, your source code will be by far the smallest part of the overall application in the vast majority of projects. Second, the analysis methods in this area are still at a very early stage of development.\nWhen using SAST, it makes a lot more sense to deal with all the other components first. Therefore, all binary files of the entire application environment are meant here. This also includes all elements that play a role in the development, such as the CI server and the other tools used.\nDAST - Dynamic Application Security Testing\nThe DAST analysis method does precisely the opposite in comparison to the SAST method. Here the application is viewed as a black box. As soon as an executable version of the application is available, it is attacked through automatically executed cyber attacks. In most cases, the general attack patterns are based on the Most Common Vulnerabilities defined by the OSWAP. Defining your attack vectors is either not provided for in the tools I know or requires very detailed knowledge in this area. For developers who do not use these tools daily and intensively, their use is limited to the predefined attack vectors. It is a good thing with this procedure that unknown security gaps can also be identified, as long as they are based on the Most Common Vulnerabilities. Some manufacturers extend this approach with AI or ML techniques. This approach using AI and ML techniques is still in a very early stage of development, and the currently available potential cannot be foreseen.\nUnfortunately, the method can only be used sensibly against the production environment, as this is the only way to identify all gaps based on configuration deficiencies. Therefore, in contrast to the SAST process, it makes no sense to use it within the CI route.\nIAST - Interactive Application Security Testing\nIAST uses software tools to evaluate application performance and identify vulnerabilities. IAST takes an \u0026ldquo;agent-like\u0026rdquo; approach; H. Agents and sensors run to continuously analyze application functions during automated tests, manual tests, or a mixture of both.\nThe process and feedback occur in real-time in the IDE, Continuous Integration (CI) environment or quality assurance or during production. The sensors have access to:\nAll source code Data and control flow System configuration data Web components Backend connection data The main difference between IAST, SAST, and DAST is that it runs inside the application.\nAccess to all static components as well as the runtime information enables a very comprehensive picture.\nIt is a combination of static and dynamic analysis. However, the part of the dynamic analysis is not a pure black-box test as it is implemented at DAST.\nSome reasons for using IAST:\nPotential problems are identified earlier, so IAST minimizes the cost of eliminating potential costs and delays.\nThis is due to a \u0026ldquo;shift left\u0026rdquo; approach, meaning it is carried out in the early stages of the project life cycle.\nSimilar to SAST, the IAST analysis provides complete lines of data-rich code so that security teams can immediately lookout for a specific bug.\nWith the wealth of information that the tool has access to, the source of vulnerabilities can be precisely identified.\nUnlike other dynamic software tests, IAST can be easily integrated into Continuous Integration and Deployment (CI / CD) pipelines.\nThe evaluations take place in real-time in the production environment.\nOn the other hand:\nIAST tools can slow down the operation of the application.\nThis is based on the fact that the agents change the bytecode themselves. This leads to a lower performance of the overall system.\nThe change itself can, of course, also lead to problems in the production environment.\nThe use of agents naturally also represents a potential source of danger since these agents can also be compromised.\n- See Solarwinds Hack\nRASP - Runtime Application Self Protection\nRASP is the approach to secure the application from within.\nThe backup takes place at runtime and generally consists of looking for suspicious commands when they are executed.\nWhat does that mean now?\nWith the RASP approach, you can examine the entire application context on the production machine and real-time.\nHere all commands that are processed are examined for possible attack patterns.\nTherefore, this procedure aims to identify existing security gaps and attack patterns and those that are not yet known.\nHere it goes very clearly into the use of AI and ML techniques.\nRASP tools can usually be used in two operating modes.\nThe first operating mode (monitoring) is limited to observing and reporting possible attacks.\nThe second operating mode (protection) then includes implementing defensive measures in real-time and directly on the production environment.\nRASP aims to fill the gap left by application security testing and network perimeter controls.\nThe two approaches (SAST and DAST) do not have sufficient visibility into real-time data and event flows to prevent vulnerabilities from sliding through the verification process or block new threats that were overlooked during development.\nRASP is similar to Interactive Application Security Testing (IAST); the main difference is that IAST focuses on identifying vulnerabilities in the applications, and RASPs focus on protecting against cybersecurity attacks that can exploit these vulnerabilities or other attack vectors.\nThe RASP technology has the following advantages:\nFirst, RASP complements SAST and DAST with an additional layer of protection after the application is started (usually in production).\nIt can be easily applied with faster development cycles.\nUnexpected entries are checked and checked.\nIt enables you to react quickly to an attack by providing comprehensive analysis and information about the possible vulnerabilities.\nHowever, RASP tools have certain disadvantages:\nBy sitting on the application server, RASP tools can adversely affect application performance.\nIn addition, the technology (RASP) may not be compliant with regulations or internal guidelines,\nwhich allows the installation of other software or\nthe automatic blocking of services is permitted.\nThe use of this technology can also lead to the people involved believing themselves to be false security.\nThe application must also be switched offline until the vulnerability is eliminated.\nRASP is not a substitute for application security testing because it cannot provide comprehensive protection.\nWhile RASP and IAST have similar methods and uses, RASP does not perform extensive scans but instead runs as part of the application to examine traffic and activity. Both report attacks as soon as they occur; with IAST, this happens at the time of the test, while with RASP, it takes place at runtime in production.\nConclusion\nAll approaches result in a wide range of options for arming yourself against known and unknown security gaps. Here it is essential to reconcile your own needs and those of the company for the future direction.\nConclusion RASP:\nWe have seen that with RASP, there is an approach in which the application can protect itself against attacks at runtime. The permanent monitoring of your activities and the data transferred to the application enable an analysis based on the runtime environment. Here you can choose between pure monitoring or alerting and active self-protection. However, software components are added to the runtime environment with RASP approaches to manipulate the system independently. This has an impact on performance. With this approach, RASP concentrates on the detection and defence of current cyber attacks. So it analyzes the data and user behaviour in order to identify suspicious activities.\nConclusion IAST:\nThe IAST approach combines the SAST and DAST approach and is already used within the SDLC, i.e. within the development itself. This means that the IAST tools are already further \u0026ldquo;to the left\u0026rdquo; compared to the RASP tools. Another difference to the RASP tools is that IAST consists of static, dynamic and manual tests. Here it also becomes clear that IAST is more in the development phase. The combination of dynamic, static and manual tests promises a comprehensive security solution. However, one should not underestimate the complexity of the manual and dynamic security tests at this point.\nConclusion DAST:\nThe DAST approach focuses on how a hacker would approach the system. The overall system is understood like a black box, and the attacks occur without knowing the technologies used. The point here is to harden the production system against the Most Common Vulnerabilities. However, one must not forget at this point that this technology can only be used at the end of the production cycle.\nConclusion SAST:\nIf you have access to all system components, the SAST approach can be used very effectively against known security gaps and license problems. This procedure is the only guarantee that the entire tech stack can be subjected to direct control. The focus of the SAST approach is on static semantics and, in turn, is completely blind to security holes in the dynamic context. A huge advantage is that this approach can be used with the first line of source code.\nMy personal opinion:\nMy personal opinion on these different approaches is as follows:\nIf you start with DevSecOps or security in IT in general, the SAST approach makes the most sense. This is where the greatest potential threat can be eliminated with minimal effort. It is also a process that can be used in all steps of the production line. Only when all components in the system are secured against known security gaps do the following methods show their highest potential. After introducing SAST, I would raise the IAST approach and, finally, the RASP approach. This also ensures that the respective teams can grow with the task and that there are no obstacles or delays in production.\nCheers Sven\n","date":"19 July 2021","externalUrl":null,"permalink":"/posts/what-is-the-difference-between-sast-dast-iast-and-rasp/","section":"Posts","summary":" Intro:\nIn this post, we’re going to look at the differences between the various cybersecurity defence techniques. Here you can identify four main groups, which we will go through briefly one after another to illustrate the advantages and disadvantages.\n","title":"What is the difference between SAST, DAST, IAST and RASP?","type":"posts"},{"content":" Intro # Again and again, we read something in the IT news about security gaps that have been found. The more severe the classification of this loophole, the more attention this information will get in the general press. Most of the time, you don\u0026rsquo;t even hear or read anything about all the security holes found that are not as well known as the SolarWinds Hack, for example. But what is the typical lifeline of such a security gap?\nhttps://youtu.be/2_b0KxUl4vs\nVulnerability was generated until found # Let\u0026rsquo;s start with the birth of a vulnerability. This birth can be done in two differently motivated ways. On the one hand, it can happen to any developer that he creates a security hole by an unfortunate combination of source code pieces. On the other hand, it can also be based on targeted manipulation. However, this has essentially no effect on the further course of the lifeline of a security vulnerability. In the following, we assume that a security hole has been created and that it is now active in some software. These can be executable programs or libraries offered that are integrated into other software projects as a dependency.\nFound until Public available # In most cases, it is not possible to understand precisely when a security hole was created. But let\u0026rsquo;s assume that there is a security hole and that it will be found. It clearly depends on which person or which team finds this weak point. This has a severe impact on the subsequent course of history. Let\u0026rsquo;s start with the fact that this vulnerability has been found by people who are interested in using this vulnerability themselves or having it exploited by other parties. Here the information is either kept under lock and key or offered for purchase at relevant places on the Internet. There are primarily financial or political motives here, which I do not want to go into here. However, at this point, it can be clearly seen that the information is passed on at this point in channels that are not generally available to the public.\nHowever, if the security gap is found by people or groups who are interested in making the knowledge about it available to the general public, various mechanisms now take effect. However, one must not forget that commercial interests will also play a role in most cases. However, the motivation is different. If the company or the project itself is affected by this vulnerability, there is usually an interest in presenting the information relatively harmless. The feared damage can even lead to the security gap being fixed, but knowledge about it further is hidden. This approach is to be viewed as critical, as it must be assumed that there will also be other groups or people who will gain this knowledge.\nBut let\u0026rsquo;s assume that people who are not directly involved in the affected components found the information about the vulnerability. In most cases, this is the motivation to sell knowledge of the vulnerability. In addition to the affected projects or products, there are also providers of the vulnerability databases. These companies have a direct and obvious interest in acquiring this knowledge. But to which company will the finder of the vulnerability sell this knowledge? Here it can be assumed that there is a very high probability that it will be the company that pays the better price. This has another side effect that affects the classification of the vulnerability. Many vulnerabilities are given an assessment using the CVSS. Here, the base value is made by different people. Different people will have other personal interests here, which will then also be reflected in this value.\nHere I refer to the blog post about CVSS - explained. https://svenruppert.com/2021/04/07/cvss-explained-the-basics/\nRegardless of the detours via which the knowledge comes to the vulnerabilities databases. Only when the information has reached one of these points can one assume that this knowledge will be available to the general public over time.\nPublic available until consumable # However, one fact can be seen very clearly at this point. Regardless of which provider of vulnerabilities you choose, there will only ever be a subset of all known vulnerabilities in this data set. As an end consumer, there is only one sensible way to go here. Instead of contacting the providers directly, you should rely on integrators. This refers to services that integrate various sources themselves and then offer them processed and merged. It is also essential that the findings are processed so that further processing by machines is possible. This means that the meta-information such as the CVE or the CVSS value is supplied. This is the only way other programs can work with this information. The CVSS value is given as an example. This is used, for example, in CI environments to interrupt further processing when a particular threshold value is reached. Only when the information is prepared in this way and is available to the end-user can this information be consumable. Since the information generally represents a considerable financial value, it can be assumed in the vast majority of cases that the commercial providers of such data sets will have access to updated information more quickly than freely available data collections.\nConsumable until running in Production # If the information can now be consumed, i.e. processed with the tools used in software development, the storyline begins in your projects. Whichever provider you have decided on, this information is available from a particular point in time, and you can now react to it yourself. The requirement is that the necessary changes are activated in production as quickly as possible. This is the only way to avoid the potential damage that can result from this security vulnerability. This results in various requirements for your software development processes. The most obvious need is the throughput times. Only those who have implemented a high degree of automation can enable short response times in the delivery processes. It is also an advantage if the team concerned can make the necessary decisions themselves and quickly. Lengthy approval processes are annoying at this point and can also cause extensive damage to the company.\nAnother point that can release high potential is the provision of safety-critical information in all production stages involved. The earlier the data is taken into account in production, the lower the cost of removing security gaps. We\u0026rsquo;ll come back to that in more detail when the shift-left topic is discussed.\nAnother question that arises is that of the effective mechanisms against vulnerabilities.\nTestCoverage is your safety-belt; try Mutation Testing # The best knowledge of security gaps is of no use if this knowledge cannot be put to use. But what tools do you have in software development to take efficient action against known security gaps? Here I would like to highlight one metric in particular: the test coverage of your own source code parts. If you have strong test coverage, you can make changes to the system and rely on the test suite. If a smooth test of all affected system components has taken place, nothing stands in the way of making the software available from a technical point of view.\nBut let\u0026rsquo;s take a step back and take a closer look at the situation. In most cases, changing the version of the same dependency in use will remove known vulnerabilities. This means that efficient version management gives you the agility you need to be able to react quickly. In very few cases, the affected components have to be replaced by semantic equivalents from other manufacturers. And to classify the new composition of versions of the same ingredients as valid, strong test coverage is required. Manual tests would go far beyond the time frame and cannot be carried out with the same quality in every run. But what is strong test coverage?\nI use the technique of mutation testing. This gives you more concrete test coverage than is usually the case with the conventional line or branch coverage. Unfortunately, a complete description of this procedure is not possible at this point.\nHowever, if you want to learn more about mutation testing, visit the following URL. This video explains the theoretical and practical part of the mutation test for Java and Kotlin.\nhttps://www.youtube.com/watch?v=6Vej7YEOF8g\nThe need for a single point that understands all repo-types # If we now assume that we want to search for known security vulnerabilities in the development and the operation of the software, we need a place where we can carry out the search processes. Different areas are suitable here. However, there is a requirement that enables an efficient scan across all technologies used. We are talking about a logical central point through which all binaries must go. I don\u0026rsquo;t just mean the jar files declared as dependencies, but also all other files such as the Debian packages or Docker images. Artifactory is suitable for this as a central hub as it supports pretty much all package managers at one point. Because you have knowledge of the individual files and have the metadata, the following things can also be evaluated.\nFirst of all, it is not only possible to capture the direct dependencies. Knowing the structures of the package managers used means that all indirect dependencies are also known. Second, it is possible to receive cross-technology evaluations. This means the full impact graph to capture the practical meaning of the individual vulnerabilities. The tool from JFrog that can give an overview here is JFrog Xray, and we are directly connected to JFrog Artifactory. Whichever tool you choose, it is crucial that you don\u0026rsquo;t just scan one technology layer. Only with a comprehensive look at the entire tech stack can one guarantee that there are no known security gaps, either directly or indirectly, in production.\nConclusion # Now we come to a conclusion. We have seen that we have little influence on most sections of a typical lifeline of an IT security vulnerability. Actually, there are only two sections that we can influence directly. On the one hand, it is the quickest and most comprehensive possible access to reliable security databases. Here it is essential that you not only entrust yourself to one provider but rather concentrate on so-called “mergers” or “aggregators”. The use of such supersets can compensate for the economically motivated vagueness of the individual providers. I named JFrog Xray as an example of such an aggregator. The second section of a typical lifeline is in your own home. That means, as soon as you know about a security gap, you have to act yourself—robust automation and a well-coordinated DevSecOps team help here. We will deal with precisely this section from the “security” perspective in another blogpost. Here, however, we had already seen that strong test coverage is one of the critical elements in the fight against vulnerabilities. Here I would like to refer again to the test method “Mutation Testing”, which is a very effective tool in TDD.\nAnd what can I do right now?\nAou can, of course, take a look at my YouTube channel and find out a little more about the topic there. I would be delighted to welcome you as my new subscriber. Thanks!\nhttps://www.youtube.com/@OutdoorNerd\nHappy Coding\nSven\n","date":"25 June 2021","externalUrl":null,"permalink":"/posts/the-lifeline-of-a-vulnerability/","section":"Posts","summary":" Intro # Again and again, we read something in the IT news about security gaps that have been found. The more severe the classification of this loophole, the more attention this information will get in the general press. Most of the time, you don’t even hear or read anything about all the security holes found that are not as well known as the SolarWinds Hack, for example. But what is the typical lifeline of such a security gap?\n","title":"The Lifeline of a Vulnerability","type":"posts"},{"content":"In this episode, I\u0026rsquo;ll show you how to make a comfortable bushcrafting chair out of a few sticks and a little string.\nStep 1 - Finding suitable wood # For this seat, we need three wooden poles. These must be stable enough to bear the weight of whoever wants to sit on them. When choosing the pieces of wood, you should, as always, make sure that only dead wood is used. In addition to the aspect of not damaging living trees, it also has a convenient background. With fresh wood, you always have to expect that there is still some moisture and resin in the workpiece and leak out in different places over time. In my search, I opted for secluded beech wood. All bars have a diameter of approx. 10cm are dry and sufficiently stable to hold my weight individually. Of course, rotten wood is not advisable.\nhttps://youtu.be/Jm8wkgsIeh8\n# Step 2 - cutting the workpieces\nFor the two outer poles, two branches with a length of approx. 1.50 m to 1.80 m are required. With these two bars, it doesn\u0026rsquo;t matter that they are straight. Here you can also use slightly crooked workpieces. A wooden rod with a length of 1m to 1.50m can be used for the seat itself. This bar can be a little thicker so that the later seating comfort is a little more fabulous.\n# Step 3 - smoothing the workpieces\nAfter the branches have all been sawn to the correct length, all protruding branches should be removed. In this step, you can remove the branches with a hatchet and any remaining bark remnants. Removing the bark residues ensures that the pieces of wood used can withstand the weather for longer. Not only does moisture collect under the bark, but also insects will quickly find a new home here. When you have now sawn, debarked and smoothed all three pieces of wood, the assembly begins.\nStep 4 - frame construction # The frame structure itself consists of two long rods. For this purpose, these are tied together on one side, similar to a tripod. For this, I use a rope made from natural products such as hemp. Of course, you can also use paracord. However, I advise against it if you do not want to dismantle this seat immediately after use. If the bushcrafting seat remains in the forest, I ask you not to use plastic-containing cords. The connection itself is kept quite simple and essentially only consists of a few loops and a final knot.\n# Step 5 - the seat\nAfter the two long poles have been connected, this frame should be set up once. On the one hand, you can see whether the newly created connection is holding. On the other hand, you can use the third rod to try out where it should be attached. Once you have decided on a position, you can begin to attach the last piece. To do this, we put the frame construction back on the floor. After the piece has been placed on the two legs of the frame structure, both sides can be connected. It would be best if you made sure that the connections are sufficiently stable and resilient, as this is where the highest loads occur.\n# Step 6 - commissioning\nAfter all, connections have been made; you can set up the construction and try it out right away. The two long poles are leaned against a tree trunk with the upper part. However, the angle to the tree trunk also determines the seat height and stability.\nAnd the bushcraft chair is ready. The entire construction should be set up in about 15 minutes after the workpieces have been found. The short construction time makes this project ideal for a shared adventure with children.\nHave fun with the reproduction.\n","date":"26 May 2021","externalUrl":null,"permalink":"/posts/howto-building-a-bushcrafting-seat-in-the-woods/","section":"Posts","summary":"In this episode, I’ll show you how to make a comfortable bushcrafting chair out of a few sticks and a little string.\nStep 1 - Finding suitable wood # For this seat, we need three wooden poles. These must be stable enough to bear the weight of whoever wants to sit on them. When choosing the pieces of wood, you should, as always, make sure that only dead wood is used. In addition to the aspect of not damaging living trees, it also has a convenient background. With fresh wood, you always have to expect that there is still some moisture and resin in the workpiece and leak out in different places over time. In my search, I opted for secluded beech wood. All bars have a diameter of approx. 10cm are dry and sufficiently stable to hold my weight individually. Of course, rotten wood is not advisable.\n","title":"Howto Building a Bushcrafting Seat in the woods","type":"posts"},{"content":" Intro # What is the Common Vulnerability Scoring System short called CVSS, who is behind it, what are we doing with it and what a CVSS Value means for you? I will explain how a CVSS Score is calculated, what the different elements of it mean and what are the differences between the different CVSS versions.\nThe Basic Idea Of CVSS # The basic idea behind CVSS is to provide a general classification of the severity of a security vulnerability. This is about the classification and evaluation of weak points. But what does the abbreviation CVSS mean?\nWhat does the abbreviation CVSS mean?\nThe letters stand for the words: Common Vulnerability Scoring System. That means something like a general vulnerability rating system. Here, the weak points found are evaluated from various points of view. These elements are weighted against each other so that a standardized number between 0 and 10 is obtained at the end.\nFor what do you need such a rating system?\nA rating system that provides us with a standardized number allows us to evaluate different weak points abstractly and derive follow-up actions from them. The focus here is on standardizing the handling of these weak points. As a result, you can define actions based on the value ranges. Here I mean processes in the value creation sources that are affected by this weak point.\nWhat is the basic structure of this assessment?\nIn principle, CVSS can be described so that the probability and the maximum possible damage are related using predefined factors. The basic formula for this is: risk = probability of occurrence x damage\nhttps://youtu.be/CZrS_UFT37Y\nThe Basic Values from 0..10 # The evaluation in the CVSS is based on various criteria and is called \u0026ldquo;metrics\u0026rdquo;. For each metric, one or more values selected from a firmly defined selection option. This selection then results in a value between 0.0 and 10.0. Where 0 is the lowest and 10 is the highest risk value. The entire range of values ​​is then subdivided into groups and are labelled \u0026ldquo;None \u0026ldquo;, \u0026ldquo;Low \u0026ldquo;, \u0026ldquo;Medium \u0026ldquo;, \u0026ldquo;High \u0026ldquo;, and \u0026ldquo;Critical \u0026ldquo;. These metrics are divided into three areas that are weighted differently from one another. These are the areas \u0026ldquo;Basic Metrics\u0026rdquo;, \u0026ldquo;Temporal Metrics\u0026rdquo;, and \u0026ldquo;Environmental Metrics\u0026rdquo;. Here, different aspects are queried in each area, which must be assigned a single value. The weighting among each other and the subsequent composition of the three group values ​​gives the final result.\nHowever, all component values ​​that lead to this result are always supplied. This behaviour ensures that there is transparency at all times about how these values ​​originally came about. Next, the three sub-areas will be explained individually in detail.\nThe Basic Metrics # The basic metrics form the foundation of this rating system. The aim is to record the technical details of the vulnerability that will not change over time. You can imagine it to be an assessment that is independent of other changing elements. Different parties can carry out the calculation of the base value. It can be done by the discoverer, the manufacturer of the project or product concerned or by a party (CERT) charged with eliminating this weak point. One can imagine that, based on this initial decision, the value itself will turn out differently since the individual groups pursue different goals.\nnecessary requirements:\nThe base value evaluates the prerequisites that are necessary for a successful attack via this security gap. This is, for example, the distinction between whether a user account must be available on the target system for an attack that is used in the course of the attack or whether the system can be compromised without the knowledge about a system user. It also plays a significant role in whether a system is vulnerable over the Internet or whether physical access to the affected component is required.\nComplexity of the attack:\nThe base value should also reflect how complex the attack is to carry out. In this case, the complexity relates to the necessary technical steps and includes assessing whether the interaction with a regular user is essential. Is it sufficient to encourage any user to interact, or does this user have to belong to a specific system group (e.g. administrator)? At this point, it is already evident that the assessment of a new vulnerability requires exact knowledge of this vulnerability and the systems concerned. The correct classification is not a trivial process.\nAssessment of the damage:\nThe basic metrics also take into account the damage that this attack could cause to the affected component. This means the possibilities to extract the data from the system, manipulate it, or completely prevent the system\u0026rsquo;s use. One speaks here of the three areas;\nConfidentiality Integrity Availability However, you have to be careful here concerning the weighting of these possibilities. In one case, it can be worse when data is stolen than it is changed. In another case, the unusability of a component can be the worst damage to be assumed.\nScope-Metric:\nThe scope metric has also been available since CVSS version 3.0. This metric looks at the effects of an affected component on other system components. For example, one can imagine that a compromised element in a virtualized environment enables access to the carrier system. A successful change of this scope represents a greater risk for the overall system and is therefore also evaluated using this factor. This point alone clearly shows that the interpretation of the values also requires adjusting to one\u0026rsquo;s situation. And so we come to the \u0026ldquo;temporal\u0026rdquo; and \u0026ldquo;environment\u0026rdquo; metrics.\nThe Temporal Metrics # The time-dependent components of the vulnerability assessment are brought together in the group of temporal metrics. The peculiarity at this point is that the base value can be reduced by the temporal components only. The initial rating is intended to represent the worst-case scenario.\nThis has both advantages and disadvantages if you bear in mind that it is during the initial assessment of a vulnerability that can give very different interests. At this point, there are two things that need to be highlighted;\n_1) Which factors influence the temporal metrics? _\nThe elements that change over time influence the \u0026ldquo;Temporal Metrics\u0026rdquo;.\nOn the one hand, this refers to changes concerning the availability of tools that support the exploitation of the vulnerability. These can be exploits or step-by-step instructions. A distinction must be made whether a chess point is theoretical or whether a manufacturer has officially confirmed it. All of these events change the base value.\n2) Influence of the initial evaluation?\nThe influence on the initial evaluation comes about through external framework conditions. These take place over an undefined time frame and are not relevant for the actual basic assessment. Even if an exploit is already in circulation during the base values survey, this knowledge will not be included in the primary assessment. However, the base value can only be reduced by the temporal metrics. This approach takes a little getting used to and is often the subject of criticism. The reason why you decided on it is understandable from the theoretical point of view. The base value is intended to denote the most excellent possible damage.\nAnd this is where a conflict arises. The person or group who has found a security gap tries to set the base value as high as possible. A highly critical loophole is better to sell and better exploited in the media. The reputation of the person/group who found this gap increases as a result. The affected company or the affected project is interested in exactly the opposite assessment. It, therefore, depends on who finds the security gap, how the recycling should take place and by which body the first evaluation is carried out. The only offsetting component is the Environmental Metrics.\nThe Environmental Metrics # In the case of environmental metrics, the own system landscape is set in relation to the security gap. This means that the evaluation is adjusted based on the real situation. In contrast to Temporal Metrics, Environmental Metrics can correct the base value in both directions. The environment can therefore lead to a higher classification and must also be constantly adapted to your own changes. The combination now gives rise to purely practical questions. Let\u0026rsquo;s assume that there is a patch from the manufacturer for a security hole. The mere presence of this modification leads to a lowering of the total value in the Temporal Metrics. However, as long as the patch has not been activated in your own systems, the overall value must be drastically corrected upwards again via the Environmental Metrics. Why did I say at this point that the value has to be increased drastically? As soon as a patch is available, this patch can be used to better understand the security gap and its effects. The attacker has more and more detailed information that can be used. This reduces the resistance of the not yet hardened systems.\nThe Final Score # At the end of an evaluation, the final score is obtained, calculated from the three previously mentioned values. At this point, I will not explain the calculation details; I have a separate post on this. The resulting value is then assigned to a value group. But there is one more point that is very important to me personally. In many cases, I see that the final score is simply carried over. The individual adjustments utilizing the environmental score do not take place. Instead, the value one to one is adopted as the risk assessment. In many cases, this behaviour leads to a dangerous evaluation which is incorrect for the overall system concerned.\nConclusion # We come to the management summary. With the CVSS we have a value system for evaluating security gaps in software. Since there are no alternatives, the system has been in use worldwide for over ten years and is constantly being developed, it is a defacto standard. The evaluation consists of three components. First, there is the basic score, which is there to depict a purely technical worst-case scenario. The second component is the evaluation of the time-dependent corrections based on external influences. This is about the evaluation of whether there are further findings, tools or patches for this security gap. The peculiarity of this point is that the base score can only be reduced with this value. The third component is the assessment of your own system environment with regard to this weak point. With this consideration, the security gap is adjusted in relation to the real situation on site. This value can lower or higher the base rating. Last but not least, an overall evaluation is made from these three values, which results in a number between 0.0 to 10.0.\nThis final value can be used to control your own actions to defend against the security gap in the overall context. At first glance, everything is quite abstract. It takes some practice to get a feel for the figures it contains.\nThe CVSS can only develop its full effect if it deals more deeply with this matter within your own system.\nI will explain the underlying calculations in detail in further posts.\n","date":"7 April 2021","externalUrl":null,"permalink":"/posts/cvss-explained-the-basics/","section":"Posts","summary":" Intro # What is the Common Vulnerability Scoring System short called CVSS, who is behind it, what are we doing with it and what a CVSS Value means for you? I will explain how a CVSS Score is calculated, what the different elements of it mean and what are the differences between the different CVSS versions.\n","title":"CVSS - explained - the Basics","type":"posts"},{"content":" # When I\u0026rsquo;m out in the forest, I always enjoy having some tools with me. The backpack is certainly a little heavier, but there are countless possibilities. My son was there again on one of my last tours. At the time I write this blog post (early 2021), he is ten years old. It\u0026rsquo;s always lovely to see the enthusiasm with which new things are tried.\nWe had set ourselves the goal of building a bank. We had an axe, a saw and a paracord with us as tools.\nTo find a suitable workpiece, forest areas in which there are deciduous trees are suitable. Conifers are usually not very ideal because they contain much resin. Now and then, you will find places where some medium-sized trees such as birch, beech or oak have been fallen due to the last storm, are relocated. Of course, you have to be careful in such places whether the tree trunks are still under tension. We found just such a place, and we immediately started to sift through the wood. Since the logs had fallen for some time, it was safe to move between them.\nhttps://youtu.be/moCUh-6mhdw\nWhat should a suitable workpiece look like? # Trunks that do not entirely rest on the ground are best. These pieces are usually dry and have a firm consistency. If you tap the wood with your fingers, you can hear a bright sound. If the sound is dull or if the wood is a bit musty, it is usually not worth sawing out a piece.\nThe length can be chosen freely and is only limited by the total weight you can carry yourself. Depending on the length, the trunk section must, of course, have a suitable thickness in diameter. After all, this piece of wood is supposed to withstand the weight of the people sitting on it. The comfort of the seat is, of course, greater the deeper the seat is. If in doubt, choose a larger diameter.\nIn our case, we have chosen a reasonably short piece of about one meter. The diameter is about 25cm and therefore sufficient for the expected weight.\nSawing the sections # To sit comfortably on the bench, one long side is now removed from the tree trunk piece. The aim is to get a flat surface over the entire width of the beam. So that the stability does not suffer too much, it is advisable not to split the trunk directly in the middle. It is possible to saw off a piece over the entire length, but it is quite a laborious task. I prefer to saw across the workpiece with a saw at intervals of about a hand\u0026rsquo;s breadth. The cut should be so deep that it corresponds to the portion to be removed.\nManufacture the seat # When the trunk has now been sawed, the tree trunk\u0026rsquo;s unneeded part can be removed with the axe. It is best to start with a middle segment. I hold the workpiece upright with my left hand and then use the axe with my right hand. By sawing, the pieces come off quite easily. When all segments have now been removed with the axe, the fine work can begin. With the axe or knife, you can now work on the seat as finely as you would like to do it.\nFind and saw wood for tripods # The main piece is now finished. Now the pieces of wood are missing with which you can make the two tripods you need. Six sticks of the same length, if possible, with a diameter of about 4 cm or a little more are needed. Of course, you have to take the total weight into account here too.\nIn our case, sticks a little more than 40cm long and about 4cm in diameter were sufficient. Again, dry and stable dead wood from deciduous trees should be used.\nTie a tripod # To tie a tripod out of three sticks, you need a little bit of Paracord. A 2m long piece of string should be sufficient. I am placing the three sticks next to each other so that there is no space between them. The cord is attached to one of the outer sticks in the middle and then wrapped around all sticks. It would be best if you made sure that the bars stay tightly together. Towards the end, you can wrap the end piece between the sticks a few more times. The end of the Paracord should then be knotted tightly. The tripod is now ready and can be set up.\nAssemble the bench # Time for the last work step now. Take the first workpiece and place it on the two tripods. After a slight fine adjustment, the bench should be ready.\nInstallation # The high point is to try out the bank and see if everything will hold up. All work steps can be carried out in about 30 minutes and are ideally suited as a small leisure project. Besides, there is, of course, the time to find suitable wood is not included.\n","date":"5 April 2021","externalUrl":null,"permalink":"/posts/build-a-small-bushcraft-bench-from-a-log/","section":"Posts","summary":" # When I’m out in the forest, I always enjoy having some tools with me. The backpack is certainly a little heavier, but there are countless possibilities. My son was there again on one of my last tours. At the time I write this blog post (early 2021), he is ten years old. It’s always lovely to see the enthusiasm with which new things are tried.\nWe had set ourselves the goal of building a bank. We had an axe, a saw and a paracord with us as tools.\nTo find a suitable workpiece, forest areas in which there are deciduous trees are suitable. Conifers are usually not very ideal because they contain much resin. Now and then, you will find places where some medium-sized trees such as birch, beech or oak have been fallen due to the last storm, are relocated. Of course, you have to be careful in such places whether the tree trunks are still under tension. We found just such a place, and we immediately started to sift through the wood. Since the logs had fallen for some time, it was safe to move between them.\n","title":"build a small Bushcraft bench from a log","type":"posts"},{"content":"Today we\u0026rsquo;re going to look at how to make a rocket stove with an axe and a saw.\nThe construction does not require any other materials such as wire or the like.\nThis is a variant that can be used to boil water yourself.\nEven in wet weather, this variant works very well because, with the correct selection of the workpiece, the inner dry wood is used to generate sufficient embers.\nIn this construction, the physical effect is used, in which hot air rises faster than the colder ambient air. This is, among other things, the reason why this variant works better the colder the ambient temperature is.\nhttps://youtu.be/zWxT-G__UZ4\nSearch / select wood # The first step is to choose a suitable piece of wood. There are a few things to consider here.\nOnly one tree that has already died should be used for this work. There are several reasons for this.\nOn the one hand, the principle \u0026ldquo;Leave no trace\u0026rdquo; applies to me. So, avoid all traces as much as possible.\nTherefore, damaging or even cutting down trees that are still alive is an absolute NO-GO for me.\nBut there are also purely practical reasons to concentrate mainly on deadwood when searching.\nMost woods are much better to use as fuel than fresh and, therefore, still damp wood.\nExceptions here are woods with a very high proportion of resin. Many conifers are part of it, but birch is also an excellent raw material for a fire. Birch burns very well, even when it is wet. This is also one of the reasons why birch bark is very popular as a tinder material.\n(But I\u0026rsquo;ll show that in more detail in another video)\nHowever, there is one more thing to keep in mind. If you want to use this rocket stove for cooking, you should avoid wood that is too resinous. During the incineration, many particles then settle on the cookware, which then has to be removed with incredible difficulty. So think a little about the work that follows.\nWood that lies on the floor is usually more humid, possibly even slightly rotten. These woods can no longer be used and have a poor calorific value, and are usually the basis of many insects\u0026rsquo; lives.\nIt is best if the part of the tree trunk is suspended in the air, i.e. not touching the ground. If you knock on wood, you can already tell from the sound how much moisture is still to be expected in the wood or how strong the decomposition process has already progressed. A bright sound is usually auspicious.\nLet\u0026rsquo;s come to the size that is well suited for this project.\nI prefer woods that are no more than the length of the palm of my hand.\nThere are several reasons for this. Firstly, these are easier to edit in the following work steps.\nSecond, these sizes can still be easily edited with handy tools. For thicker workpieces, you usually have to provide makeshift tools.\n(How to split very thick tree trunks with a small axe, which I will describe in detail in another video.)\nSaw off wood # It has been shown to me that I prefer to use pieces between 30-50 cm long in terms of length.\nThey have the advantage that they are stable in the fire for a long time, can be quickly processed and do not burn too long. The burning time is sufficient for preparing a meal for up to two people, followed by coffee and warm up a little while the coffee is still being enjoyed. The burning time, however, is heavily dependent on the type of wood used and its quality. For sawing myself, I use a saw on tours that has a relatively long blade. Indeed, shorter saw blades can also be used, but I prefer to carry a few grams more with me and then have more comfort when working on the wood.\nSplit / split wood # As soon as the workpiece has been found and sawed out, you can start splitting.\nAn old tree stump that can be used as a base is suitable for this.\nThis has the advantage that the work can also be carried out quite quietly.\nWho would want to reveal their position to all hunters and foresters with loud noises?\nStones are only conditionally suitable as a base. Here you have to reckon that the axe hits the stony surface while working with the blade and thus quickly loses its sharpness.\nThe piece of wood must now be split lengthways into three or four equal parts, if possible.\nThe trunk must be divided along its entire length.\nIt is also crucial that the parts are as thick as possible so that all aspects burn at the same speed later during usage.\nRemove insides # Now that we have three or four pieces, the chimney can be carved out. This is done on the inside edge, lifted some chips. The result is a free space in the middle. This achieved that an internal tube is created that can be used as a stove pipe. Ultimately, the point is to enable the hot air to rise in a targeted manner. This makes a draft that will constantly draw in fresh air from below. The chimney effect has already been achieved. When all parts are put back together, you can look through the length of the workpiece once.\nThe work itself does not have to be carried out too carefully. Likewise, not too much material should be removed, as it will only shorten the burning time. A reasonably small opening is enough for a good draft. I use my thumb as a measure of the diameter.\nCut the combustion chamber # There are now different ways to operate the Rocket Stove. I mostly cut a small combustion chamber into the bottom. For this purpose, a notch is engraved on the underside of a side part.\nFor this, you can do the rough preparatory work with the saw by cutting a triangle. Then the opening is worked out a little more with the axe. But you can do that at will.\nIt should just be such that the opening is enough to add fuel. This facilitates lighting during the first few minutes to light a sufficiently large flame.\nassembling and installation # A relatively straight base is required to operate the furnace. There the parts are placed next to each other.\nMost of the solutions I see with others use a piece of wire to hold the pieces together.\nFirst of all, you must first have carried such a wire with you and, secondly, the wire must not have been left behind at the place later.\nAttach side supports # I use material from the surrounding area to support the parts. A few small branches that are anchored in the ground to hold the side parts are sufficient. And you are done with the construction and can start operating. As it burns down, the small sticks are simply moved along the sides of the fire.\nYou could also sharpen the underside of the individual parts. This means that you can even anchor the stove on the floor. This even saves you the side supports.\nSo, leave the wire at home and use a couple of small sticks.\nCollect birch bark # Birch bark is ideal as a tinder material. Usually, there are individual birch trees in many places in the forests, a pioneer plant. Some birches have some areas where the bark has already peeled off. It is better to use dead birch trees. You can then peel off the bark from these trees with a knife or axe. The birch wood can also be used as it contains quite a lot of essential oils, which enables it to ignite when it is moist.\nPrepare the tinder # Now we come to the operation of the Rocket Stove. For this, we need tinder to get the flame so big that we can light the wood with it. For this, you can use the bark of the birch to light a flame with a few sparks from a fire steel. Chop up the birch bark a little and place it on top of additional tinder material. Old bark is suitable as a base to get the glowing nest into the combustion chamber. If you have fat wood, you can, of course, also use it.\nKindle # As soon as you have lit the fire on the bark underneath, you can use a stick to push the embers into the combustion chamber. The chimney effect should set in immediately and ensure a reasonably constant flow of fresh air. After a short time, the fire begins to rise along the chimney pipe. The resulting heat flows out of the opening and can be used immediately to heat food or water.\nHowever, it is not advisable to place the mug directly on the chimney opening, as this interrupts airflow. Much smoke will usually be the result; two small sticks under the cup will solve the problem.\nConclusion # We are now able to manufacture a rocket stove ourselves with minimal effort. This gives you a controlled fireplace that is well suited to heating food and water in damp weather. It is also recommended to use it as a heat source near a warehouse. The wood consumption is meagre with good heat yield at the same time.\nTo extinguish the flame, you can pull the individual parts of the Rocket-Stove apart and press the embers on the floor. The flight of sparks is easy to control with this type of fireplace. In an emergency, the fire can be annihilated quickly with a bit of water if allowed to run directly into the chimney opening. The embers are, of course, not yet extinguished.\n","date":"13 March 2021","externalUrl":null,"permalink":"/posts/building-a-rocket-stove-with-axe-and-saw-only/","section":"Posts","summary":"Today we’re going to look at how to make a rocket stove with an axe and a saw.\nThe construction does not require any other materials such as wire or the like.\nThis is a variant that can be used to boil water yourself.\nEven in wet weather, this variant works very well because, with the correct selection of the workpiece, the inner dry wood is used to generate sufficient embers.\nIn this construction, the physical effect is used, in which hot air rises faster than the colder ambient air. This is, among other things, the reason why this variant works better the colder the ambient temperature is.\n","title":"Building a Rocket-Stove with Axe and Saw only","type":"posts"},{"content":" Builder-Pattern # The book from the \u0026ldquo;gang of four\u0026rdquo; is part of the essential reading in just about every computer science branch. The basic patterns are described and grouped to get a good start on the topic of design patterns. But how does it look later in use?\nHere we will take a closer look at one pattern and expand it.\nhttps://youtu.be/n7X_XT55-jw\nThe Pattern - Builder # The builder pattern is currently enjoying increasing popularity as it allows you to build a fluent API.\nIt is also lovely that an IDE can generate this pattern quite quickly. But how about using this design pattern in daily life?\nThe basic builder pattern # Let\u0026rsquo;s start with the basic pattern, the initial version with which we have already gained all our experience.\nFor example, I\u0026rsquo;ll take a Car class with the Engine and List \u0026lt; Wheels\u0026gt; attributes. A car\u0026rsquo;s description is certainly not very precise, but it is enough to demonstrate some specific builder-pattern behaviours.\nNow let\u0026rsquo;s start with the Car class.\npublic class Car { private Engine engine; private List wheelList; //SNIPP } At this point, I leave out the get and set methods in this listing. If you generate a builder for this, you get something like the following.\npublic static final class Builder { private Engine engine; private List\u0026lt;Wheel\u0026gt; wheelList; private Builder() { } public Builder withEngine(Engine engine) { this.engine = engine; return this; } public Builder withWheelList(List\u0026lt;Wheel\u0026gt; wheelList) { this.wheelList = wheelList; return this; } public Car build() { return new Car(this); } } Here the builder is implemented as a static inner class. The constructor of the \u0026ldquo;Car\u0026rdquo; class has also been modified.\nprivate Car(Builder builder) { setEngine(builder.engine); wheelList = builder.wheelList; } On the one hand, there has been a change from public to private , and on the other hand, an instance of the builder has been added as a method parameter.\nCar car = Car.newBuilder() .withEngine(engine) .withWheelList(wheels) An example - the car # If you now work with the Builder Pattern, you get to the point where you have to build complex objects. Let us now extend our example by looking at the remaining attributes of the Car class.\npublic class Car { private Engine engine; private List\u0026lt;Wheel\u0026gt; wheelList; } public class Engine { private int power; private int type; } public class Wheel { private int size; private int type; private int colour; } Now you can have a corresponding builder generated for each of these classes. If you stick to the basic pattern, it looks something like this for the class Wheel :\npublic static final class Builder { private int size; private int type; private int colour; private Builder() {} public Builder withSize(int size) { this.size = size; return this; } public Builder withType(int type) { this.type = type; return this; } public Builder withColour(int colour) { this.colour = colour; return this; } public Wheel build() { return new Wheel(this); } } But what does it look like if you want to create an instance of the class Car? For each complex attribute of Car , we will create an instance using the builder. The resulting source code is quite extensive; at first glance, there was no reduction in volume or complexity.\npublic class Main { public static void main(String[] args) { Engine engine = Engine.newBuilder().withPower(100).withType(5).build(); Wheel wheel1 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); Wheel wheel2 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); Wheel wheel3 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); List\u0026lt;Wheel\u0026gt; wheels = new ArrayList\u0026lt;\u0026gt;(); wheels.add(wheel1); wheels.add(wheel2); wheels.add(wheel3); Car car = Car.newBuilder() .withEngine(engine) .withWheelList(wheels) .build(); System.out.println(\u0026#34;car = \u0026#34; + car); } } This source code is not very nice and by no means compact. So how can you adapt the builder pattern here so that on the one hand you have to write as little as possible by the builder himself and on the other hand you get more comfort when using it?\nWheelListBuilder # Let\u0026rsquo;s take a little detour first. To be able to raise all potentials, we have to make the source text homogeneous. This strategy enables us to recognize patterns more easily. In our example, the creation of the List is to be outsourced to a builder, a WheelListBuilder.\npublic class WheelListBuilder { public static WheelListBuilder newBuilder(){ return new WheelListBuilder(); } private WheelListBuilder() {} private List\u0026lt;Wheel\u0026gt; wheelList; public WheelListBuilder withNewList(){ this.wheelList = new ArrayList\u0026lt;\u0026gt;(); return this; } public WheelListBuilder withList(List wheelList){ this.wheelList = wheelList; return this; } public WheelListBuilder addWheel(Wheel wheel){ this.wheelList.add(wheel); return this; } public List\u0026lt;Wheel\u0026gt; build(){ //test if there are 4 instances.... return this.wheelList; } } Now our example from before looks like this:\npublic class Main { public static void main(String[] args) { Engine engine = Engine.newBuilder().withPower(100).withType(5).build(); Wheel wheel1 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); Wheel wheel2 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); Wheel wheel3 = Wheel.newBuilder().withType(2).withColour(3).withSize(4).build(); List\u0026lt;Wheel\u0026gt; wheelList = WheelListBuilder.newBuilder() .withNewList() .addWheel(wheel1) .addWheel(wheel2) .addWheel(wheel3) .build();//more robust if you add tests at build() Car car = Car.newBuilder() .withEngine(engine) .withWheelList(wheelList) .build(); System.out.println(\u0026#34;car = \u0026#34; + car); } } Next, we connect the Wheel class builder and the WheelListBuilder class. The goal is to get a fluent API so that we don\u0026rsquo;t create the instances of the Wheel class individually and then use the addWheel(Wheel w) method to WheelListBuilder need to add. It should then look like this for the developer in use:\nList wheels = wheelListBuilder .addWheel().withType(1).withSize(2).withColour(2).addWheelToList() .addWheel().withType(1).withSize(2).withColour(2).addWheelToList() .addWheel().withType(1).withSize(2).withColour(2).addWheelToList() .addWheel().withType(1).withSize(2).withColour(2).addWheelToList() .build(); So what happens here is the following: As soon as the addWheel() method is called, a new instance of the class WheelBuilder should be returned. The addWheelToList() method creates the representative of the Wheel class and adds it to the list. To do that, you have to modify the two builders involved. The addWheelToList() method is added to the WheelBuilder side. This adds the instance of the Wheel class to the WheelListBuilder and returns the instance of the WheelListBuilder class.\nprivate WheelListBuilder wheelListBuilder; public WheelListBuilder addWheelToList(){ this.wheelListBuilder.addWheel(this.build()); return this.wheelListBuilder; } On the side of the WheelListBuilder class, only the method addWheel() is added.\npublic Wheel.Builder addWheel() { Wheel.Builder builder = Wheel.newBuilder(); builder.withWheelListBuilder(this); return builder; } If we now transfer this to the other builders, we come to a pretty good result:\nCar car = Car.newBuilder() .addEngine().withPower(100).withType(5).done() .addWheels() .addWheel().withType(1).withSize(2).withColour(2).addWheelToList() .addWheel().withType(1).withSize(2).withColour(2).addWheelToList() .addWheel().withType(1).withSize(2).withColour(2).addWheelToList() .done() .build(); The NestedBuilder # So far, the builders have been modified individually by hand. However, this can be implemented generically quite easily since it is just a tree of builders.\nEvery builder knows his children and his father. The implementations required for this can be found in the NestedBuilder class. It is assumed here that the methods for setting attributes always begin with the prefix with. Since this seems to be the case with most generators for builders, no manual adjustment is necessary here. The method done() sets the result of his method build() on his father. The call is made using reflection. With this, a father knows the authority of the child. At this point, I assume that the name of the attribute is the same as the class name. We will see later how this can be achieved with different attribute names. The method withParentBuilder(..) enables the father to announce himself to his child. We have a bidirectional connection now.\npublic abstract class NestedBuilder\u0026lt;T, V\u0026gt; { public T done() { Class\u0026lt;?\u0026gt; parentClass = parent.getClass(); try { V build = this.build(); String methodname = \u0026#34;with\u0026#34; + build.getClass().getSimpleName(); Method method = parentClass.getDeclaredMethod(methodname, build.getClass()); method.invoke(parent, build); } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) { e.printStackTrace(); } return parent; } public abstract V build(); protected T parent; public \u0026lt;P extends NestedBuilder\u0026lt;T, V\u0026gt;\u0026gt; P withParentBuilder(T parent) { this.parent = parent; return (P) this; } } Now the specific methods for connecting with the children can be added to a father. There is no need to derive from NestedBuilder.\npublic class Parent { private KidA kidA; private KidB kidB; //snipp..... public static final class Builder { private KidA kidA; private KidB kidB; //snipp..... // to add manually private KidA.Builder builderKidA = KidA.newBuilder().withParentBuilder(this); private KidB.Builder builderKidB = KidB.newBuilder().withParentBuilder(this); public KidA.Builder addKidA() { return this.builderKidA; } public KidB.Builder addKidB() { return this.builderKidB; } //--------- public Parent build() { return new Parent(this); } } } And with the children, it looks like this: Here, you only have to derive from NestedBuilder.\npublic class KidA { private String note; //snipp..... public static final class Builder extends NestedBuilder\u0026lt;Parent.Builder, KidA\u0026gt; { //snipp..... } } The use is then very compact, as shown in the previous example.\npublic class Main { public static void main(String[] args) { Parent build = Parent.newBuilder() .addKidA().withNote(\u0026#34;A\u0026#34;).done() .addKidB().withNote(\u0026#34;B\u0026#34;).done() .build(); System.out.println(\u0026#34;build = \u0026#34; + build); } } Any combination is, of course, also possible. This means that a proxy can be a father and child at the same time. Nothing stands in the way of building complex structures.\npublic class Main { public static void main(String[] args) { Parent build = Parent.newBuilder() .addKidA().withNote(\u0026#34;A\u0026#34;) .addKidB().withNote(\u0026#34;B\u0026#34;).done() .done() .build(); System.out.println(\u0026#34;build = \u0026#34; + build); } } Happy Coding\n","date":"9 March 2021","externalUrl":null,"permalink":"/posts/pattern-from-the-practical-life-of-a-software-developer/","section":"Posts","summary":"Builder-Pattern # The book from the “gang of four” is part of the essential reading in just about every computer science branch. The basic patterns are described and grouped to get a good start on the topic of design patterns. But how does it look later in use?\nHere we will take a closer look at one pattern and expand it.\n","title":"Pattern from the practical life of a software developer","type":"posts"},{"content":" Intro: # Sometimes you need a small container to catch a little water, hold small things together, or only a temporarily drinking cup. Today we will look at how a makeshift cup can be made from a round wood piece with simple means. All we need is a saw, a knife and a little paracord. But one thing at a time. Let\u0026rsquo;s start by choosing the right piece of wood.\nhttps://youtu.be/8ZSfEH6Mi3U\nSelecting The Right Stick Of A Tree # There are a few things to consider when choosing the appropriate piece of wood. First of all, I would like to ask you to use dead wood whenever possible explicitly. This behaviour is not only for the reason that no trees should be damaged. Even dry deadwood has the advantage that any moisture will not affect the taste.\nUnder no circumstances should poisonous woods such as yew be used. Most yew species, such as the European yew (Taxus baccata), contain very toxic ingredients such as Taxin B. Bark, needles, and seeds are poisonous. However, the red seed coat does not contain any toxins. Cases of fatal poisoning by yew trees are known from humans, cattle and horses.\nThe use of softwood can also be unfavourable, as these woods often have a high resin content. This resin not only sticks the tools used but is also very stubborn on the skin. The resins themselves leave a nutty to very bitter taste that can be very unpleasant.\nWhen the right piece of deadwood has been found, the question of the right size comes up. Here I recommend a portion for the first attempts you can enclose with your hand if it is a drinking cup. Up to this size, the work steps can still be carried out quickly with a relatively small tool. If the pieces are too thick, a more extensive tool is needed rapidly.\nThe wood should also not have died for too long so that the structure is still firm and not decomposed by insects. If you knock on the piece of wood and make a dull sound, it may have become too damp. Elements of wood that do not touch the ground are usually more suitable, as these are dry compared to those pieces that lie directly on the ground.\nIn terms of structure, the areas that have little or no knotholes are suitable. Branches that have grown out of the trunk leave most holes in the trunk that are not conducive to a cup\u0026rsquo;s function.\nSaw The Workpiece To Size # When sawing out the workpiece, the length of the palm of the hand, including fingers, has proven to be practical for me. The longer the pieces, the more difficult it is to split them with small tools. The sawing itself should be carried out cleanly so that the edges do not splinter or break off. After the first cut, be sure to check the inside of the wood for damage from insects or fungi. If the tree is already severely damaged from the inside, further use is not recommended.\nSplit It Into Parts # The piece of wood must now be split into three or four parts. You can use an axe for this. It is also possible to use a knife and a wooden stick as a hammer. Please make sure that it is best to use a full tang/knife.Process individual parts with the knifeAs soon as the three or four parts are in place, you can start flattening the inside. The goal is to have a cavity in the middle when you put all the pieces back together later. So that you don\u0026rsquo;t accidentally edit the entire length, you can either mark it with a pen or use the saw. With the saw, you can cut the inside where the bottom of the vessel is to arise.\nYou should not work on the side walls.If you can work very precisely, it may work, but most of the time, the result is bad. Use the structure that resulted from splitting and leave it as it is. This gives excellent results in terms of water permeability.\nAssemble And Tie With Paracord # The last step is to put the individual parts back together. It is, of course, more comfortable if you have identified the individual workpieces.As soon as all parts have been brought together, you can start to wrap a piece of paracord tightly around the bottom of the cup. Complete this process with a knot. The same must then be repeated on the top of the cup. When you have everything tightly wrapped, you can start with the first operational test.\nFunction Test With Water or Coffee # Finally, you can now test the cup by filling it with water and looking for leaks. If you want, you can still seal the seams with liquid wax. In my case, I didn\u0026rsquo;t do it.Please note that only drinking water is used in the test phase with a cup that is to be used for drinking.Subsequent rinsing is not possible due to the relatively rough wooden surface.\nConclusion # We have now seen how you can make a makeshift cup in a few minutes with an axe, a saw and two pieces of paracord. It is crucial to choose the right piece of wood. Here again, the important note that you must not use poisonous woods.\nHave fun!\nCheers Sven\n","date":"9 March 2021","externalUrl":null,"permalink":"/posts/make-a-drinking-cup-from-trunk-and-paracord/","section":"Posts","summary":"Intro: # Sometimes you need a small container to catch a little water, hold small things together, or only a temporarily drinking cup. Today we will look at how a makeshift cup can be made from a round wood piece with simple means. All we need is a saw, a knife and a little paracord. But one thing at a time. Let’s start by choosing the right piece of wood.\n","title":"Make a Temporarily Drinking Cup from Wood and Paracord","type":"posts"},{"content":" Intro\nIn this article, we will look at the difference between the inheritance and delegation concepts. Or, to put it better, why I prefer delegation and why I want to emphasize this rarely-used feature in Java.\nThe Challenge\nThe challenge we face today is quite common in the field of graphic user interfaces like desktop- or web-apps. Java is widely used as the development language for both worlds, and it does not matter if we are in the classic swing, JavaFX, or the field of web frameworks like Vaadin. Explicitly, I\u0026rsquo;ve opted for a pseudo-class model in core Java, as I\u0026rsquo;d like to look at the design patterns here without any technical details.\nThe goal is to create a custom component that consists of a text input field and a button. Both elements should be displayed next to each other, i.e. in a horizontal layout. The respective components have no function in this example. I want to be here exclusively to work towards the differences between inheritance and delegation.\nTo lazy to read? Check-out my Youtube Version!\nhttps://youtu.be/xgViS-wlH7Q\nThe Base Class Model\nMostly, there are the respective essential components in a framework. In our case, it is a TextField, a button, and a horizontal or vertical layout. However, all of these components are embedded in an inheritance structure. In our case, I chose the following construction. Each component corresponds to the Component interface, for which there is an abstract implementation called AbstractComponent.\nThe class AbstractComponent contains framework-specific and technologically-based implementations. The **Button, **as well as the TextField , extend the class AbstractComponent. Layouts are usually separate and, therefore, a specialized group of components that leads in our case to an abstract class named Layout , which inherits from the class AbstractComponent.\nIn this abstract class, there are layout-specific implementations that are the same for all sorts of layouts. The implementations HorizontalLayout and VerticalLayout based on this. Altogether, this is already a quite complex initial model.\nInheritance — First Version\nIn the first version, I show a solution that I have often seen in projects. As a basis for a custom component, a base component from the framework is used as a parent. The direct inheritance from a layout is often used to structure all other internally child components on the screen. Inside the constructor, the internally required elements are generated and added to the inherited layout structure.\npublic class InputComponent extends Horizontal Layout // Layout is abstract implements HasLogger { private button button = new Button (); private TextField textField = new TextField (); public InputComponent () { addComponent (text field); addComponent (button); } public void click () { button.click (); } public void setText (String text) { textField.setText (text); } public String getText () { return textField.getText (); } } If you now look at how the component will behave during later use, it becomes visible that a derivation from a fundamental component brings its pitfalls with it.\nWhat exactly happened here? If an instance of the custom component InputComponent is now used, it can be viewed as a layout. But that is not the case here anymore; on the contrary, it is even wrong. All methods inherited from the layout implementation are also public available with this component. But you wanted to achieve something else. First of all, we wanted to reuse the existing code, provided in the component implementation HorizontalLayout.\nOn the other hand, you want a component that externally delegates only the methods for the necessary interaction, needed for the custom behaviour. In this case, the public methods from the Button and the TextField used symbolically. Besides, this component is tied to visual design that leads to possible interactions that are not part of the domain-specific behaviour of this component. This technical debt should be avoided as much as possible.\nIn practical words, general methods from the implementation of the HorizontalLayout are made visible to the outside. If somebody uses exactly these methods, and later on the parent becomes a VerticalLayout , the source code can not compile without further corrections.\npublic class MainM01 implements HasLogger { public static void main (String [] args) { var inputComponent = new InputComponent (); inputComponent.setText (\u0026#34;Hello Text M01\u0026#34;); inputComponent.click (); // critical things inputComponent.doSomethingLayoutSpecific (); inputComponent.horizontalSpecific (); inputComponent.doFrameworkSpecificThings (); } } Inheritance — Second Version # The custom component has to fit into the already existing component hierarchy from the framework. A place must be found inside the inheritance to start from; otherwise, the custom component cannot be used. But at the same time, we do not want to own specific implementation details, and neither the effort to implement basic technical requirements based on the framework needs. The point from which you split up the inheritance must be used wisely.\nPlease assume that the class AbstractComponent is what we are looking for as a start point.\nIf you derive your class from it, so you certainly have the essential features that you would like to have as a user of the framework. However, this abstraction mostly associated with the fact that also framework-specific things are to be considered. This abstract class is an internally used, fundamental element. Starting with this internal abstract class very likely leads to the need to implement internal and technical related methods. As an example, the method signature with the name doFrameworkSpecificThings() has been created and implemented with just a log message.\npublic class InputComponent extends AbstractComponent implements HasLogger { private button button = new Button (); private TextField textField = new TextField (); public InputComponent () { var layout = new HorizontalLayout (); layout.addComponent (text field); layout.addComponent (button); addComponent (layout); } public void click () { button.click (); } public void setText (String text) { textField.setText (text); } public String getText () { return textField.getText (); } // to deep into the framework for EndUser public void doFrameworkSpecificThings () { logger (). info (\u0026#34;doFrameworkSpecificThings -\u0026#34; + this.getClass (). getSimpleName ()); } } In use, such a component is already a little less dangerous. Only the internal methods that are visible on other components are accessible on this component.\npublic class MainM02 implements HasLogger { public static void main (String [] args) { var inputComponent = new InputComponent (); inputComponent.setText (\u0026#34;Hello Text M02\u0026#34;); inputComponent.click (); // critical things inputComponent.doFrameworkSpecificThings (); } } But I am not happy with this solution yet. Very often, there is no requirement for new components on the technical side. Instead, they are compositions of already existing essential elements, composed in a professional, domain-specific context.\nComposition — My Favorite\nSo what can you do at this point? The beautiful thing about the solution is that you can use it to put a wrapper around already existing components, which have been generated by inheritance. One solution may be to create a composite of type T.** Composite**\nThis class serves as an envelope for the compositions of the required components. This class can then even itself inherit from the interface Component, so those technical methods of the abstract implementation not repeated or released to the outside. The type T itself is the type to be used as the external component that holds in the composition. In our case, it is the horizontal layout. With the method getComponent(), you can access this instance if necessary.\npublic final class InputComponent extends Composite implements HasLogger { private button button = new Button (); private TextField textField = new TextField (); public InputComponent () { super (new Horizontal Layout ()); getComponent().addComponent(text field); getComponent().addComponent (button); } public void click () { button.click (); } public void setText (String text) { textField.setText (text); } public String getText () { return textField.getText (); } } Seen in this way, it is a neutral shell, but it will behave towards the outside as a minimal component since the minimum contract via the Component interface. Again, only the methods by delegation to the outside are made visible, which explicitly provided. Use is, therefore, harmless.\npublic class MainSolution { public static void main (String [] args) { var inputComponent = new InputComponent (); inputComponent.setText (\u0026#34;Hello Text M03\u0026#34;); inputComponent.click (); } } Targeted Inheritance\nLet\u0026rsquo;s conclude with what I believe is rarely used Java feature at the class level. The speech is about the keyword final.\nTo prevent an unintentional derivation, I recommend the targeted use of final classes. Thus, from this point, the unfavourable inheritance on the composition level is troublesome. Understandably, in most frameworks, no use of it was made. After all, you want to allow the user of the component Button to offer a specialized version. But at the beginning of your abstraction level, you can very well use it.\nConclusion\nAt this point, we have seen how you can achieve a more robust variant of a composition by delegation rather than inheritance. You can also use this if you are confronted with legacy source codes with this anti-pattern. It\u0026rsquo;s not always possible to clean up everything or change it to the last detail. But I hope this has given an incentive to approach this situation.\nThe source code for this example can be found on GitHub.\nCheers Sven!\n","date":"18 February 2021","externalUrl":null,"permalink":"/posts/delegation-versus-inheritance-in-graphical-user-interfaces/","section":"Posts","summary":" Intro\nIn this article, we will look at the difference between the inheritance and delegation concepts. Or, to put it better, why I prefer delegation and why I want to emphasize this rarely-used feature in Java.\n","title":"Delegation Versus Inheritance In Graphical User Interfaces","type":"posts"},{"content":"I am listing some of the outd oor equipment I use for my outdoor adventures on this page. Let me know what your experience is or what you are using.\nBackpack\nhttps://amzn.to/3jPrdUw Tasmanian Tiger TT Modular Pack, 30 L Daypack https://amzn.to/2ZiUrBV Tasmanian Tiger TT Range Pack MK II 90 + 10 https://amzn.to/3jYDt5q Kids Backpack - Vaude Hidalgo 42+8 https://amzn.to/3b9xMxf Tasmanian Tiger TT Warrior Belt MK III Equipment Belt with MOLLE System https://amzn.to/2NuUOGI Tactical Belt Adjustable MOLLE Belt https://amzn.to/37iLcpF Tasmanian Tiger TT Tac Pouch 5 https://amzn.to/3u33cOL Tasmanian Tiger TT Mesh Pouch Set VL Backpack Organiser Mesh Additional Bags https://amzn.to/3dh4AHE Tasmanian Tiger TT card holder RFID B wallet reading protection https://amzn.to/3jQ0OGh Tasmanian Tiger TT Bottle Holder, 1 L https://amzn.to/3b6lpSI Tasmanian Tiger TT L card bag, olive, 20 x 19 x 2 cm https://amzn.to/3quiZnC Tasmanian Tiger Tac 2 Radio Bag 15 x 7 x 7 cm https://amzn.to/3b7z1gC Tactical Waist Bag https://amzn.to/3jOxIXU Airsson Molle Small Bag https://amzn.to/3dd25pC Medical Bag for Outdoor https://amzn.to/3dgd2a4 Mil-Tec - Molle Belt Pouch Large https://amzn.to/37eGnO0 Tasmanian Tiger TT Rain Cover https://amzn.to/3dgg9yD Event WP Bag XL Cub Olive https://amzn.to/3anVx5E Outdoor Waterproof Storage Bags Clothes\nhttps://amzn.to/37emTcq Ridgeline Evolution Smock https://amzn.to/2N8Pf0P Ridgeline Pintail Explorer https://amzn.to/2OHkVLp Fjällräven Men\u0026rsquo;s Barents Pro Trousers https://amzn.to/2N9Uj55 Helikon-Tex Patriot Jacket - Double Fleece https://amzn.to/2MZUUGM Ridgeline Impact Fleece Top https://amzn.to/3sK1ekg Helikon-Tex Woodsman Shirt https://amzn.to/3wTDe0i FALKE functional shirt long-sleeved shirt Wool Tech virgin wool https://amzn.to/3alPSx6 Leather Gloves https://amzn.to/3aj461F Unisex Fingerless Winter Gloves https://amzn.to/3s0TJWl Helikon-Tex Range Polo Shirt TopCool Adaptive Green https://amzn.to/37i45ch Bushcraft II Fun Organic Men\u0026rsquo;s T-Shirt https://amzn.to/2N9zJ4P Meindl Sölden 670246 men\u0026rsquo;s outdoor shoes https://amzn.to/3poU3fR crampon 19 teeth claws non-slip shoe case with stainless steel chain for hiking on snow and ice snow chains https://amzn.to/2ZgK9SD Copytec #32041 Patch Bushcraft Germany Survival Outdoor Forest Prepper https://amzn.to/3bc8O0e Café Viereck ® German Army Veteran Patch Embroidered with Velcro 7 cm x 9 cm https://amzn.to/3qnfzD5 Craftigale Bushcraft Lifestyle Leder Patch (leather, natural) https://amzn.to/2NuXNPq Pinewood Unisex - Wool hat https://amzn.to/3ajWwE2 Leo Köhler Tactical Base Cap Phantomleaf WASP II https://amzn.to/3pkWUGn Tilley Endurables T3 Traditional Canvas Hat https://amzn.to/3rVsV9L Multifunctional Face Mask https://amzn.to/37jrcmZ bw new shemagh olive Navigation And Communication\nhttps://amzn.to/2ZiRz87 Garmin Foretrex 601 GPS Navigation Device https://amzn.to/3qpKr5A Planzeiger: Hilfsmittel für topografische Karten (German) https://amzn.to/2OGDHm4 10 Stück Wasserdichte Notizblock https://amzn.to/3b7hRjh Small Portable FM/AM Radio https://amzn.to/3dckwe5 80 W Foldable Solar Charger https://amzn.to/3qrQ0Rk Pack of 8 2800 mAh AA batteries with 4 Bay battery charger https://amzn.to/3bbzuyk Panasonic eneloop pro, ready-to-use Ni-MH battery, AAA micro, 4-pack https://amzn.to/3dptcxW Panasonic eneloop pro, ready-to-use Ni-MH battery, AA mignon, 4-pack https://amzn.to/3bwonAn AA 2800 mAh Rechargeable PowerOwl https://amzn.to/3s2e8u0 Midland Alan 42 DS, CB handheld radio with digital squelch and extensive accessories for all areas of use. 4W AM/FM Equipment For Food and Water\nhttps://amzn.to/2NgfuTf Katadyn Pocket Water Filter 50.000l https://amzn.to/3bbvVbg Sawyer Mini PointONE Water Filter https://amzn.to/2N7bggF Micropur Forte MF 1T - 100 (4x25) Tablets https://amzn.to/2NbR2lM 1050 ml / 600 ml Titanium Outdoor Water Bottle Leak-proof Wide Mouth https://amzn.to/3u03k1c 1100 ml + 700 ml Titanium Ultralight Military Water Bottle https://amzn.to/37h0CL2 750ml Titanium Bottle Camping Portable Water Bottle Outdoor Large Mouth https://amzn.to/3qskTVC Titanium Tasse Camping Kochgeschirrset Tragbar Becher https://amzn.to/37iQbqn Titanium Spoon Outdoor Portable Dinner Spoon Cutlery https://amzn.to/37jluBz Nalgene Everyday WH Plastic Bottle https://amzn.to/3b8y0EX 6 X Lock \u0026amp; Lock Rect 350ml Food Container https://amzn.to/3qjTLbg Pack of 100 Paper Bags https://amzn.to/3jQOGVu Honey powder 750 g https://amzn.to/3rVt0dM 1 kg Egg Powder https://amzn.to/3bc62Io BP ER Elite Emergency Food 24 x 500 g Unit https://amzn.to/3anYeEi NRG-5 Emergency Food Rations, 1 box with 24 packs x 500 g https://amzn.to/2NaTwBc Energy Cake - Box with 24 x 125 g Sleeping Bags, Tarps, and more\nhttps://amzn.to/3rW40mB DD Tarp 3m x 3m - Multicam Tarp/Basha https://amzn.to/3djsfXI Tarp 3m x 3m - Planet Gear Camping Tarp https://amzn.to/3u1hPly Tarp 3m x 3m - Pure Hang Premium Tent Tarp https://amzn.to/3dcmzPj BW Camping Mat / German Armed Forces / BW Faltmatte https://amzn.to/3dc6oS9 German armed forces wet protection cover - \u0026ldquo;Elefantenhaut\u0026rdquo; https://amzn.to/3b5yAU1 Therm-a-Rest NeoAir Xlite https://amzn.to/3qAuWrX Alpenfell Sheepskin Lambskin Brown Large Merino Sheep 120-130 cm Ecological Tanning https://amzn.to/3qmoq7O Zaloop Lambskin Sheepskin Brown Approx. 100-110 cm Ecological Tanning https://amzn.to/37gVA1n Camping Zelt Ultraleicht Mesh Zelt Moskito https://amzn.to/3rWM2QR The Friendly Swede Camping mosquito net for 1 person https://amzn.to/3ap3pUF Carinthia Defence 4 Sleeping Bag https://amzn.to/39C8AzO COVACURE hammock Fire and Cooking Equipment\nhttps://amzn.to/3qqEhST Blow Pipe Fire Tool for Outdoor Survival https://amzn.to/2OBtxTD Bush Gear Fire Steel \u0026ldquo;Black Steels\u0026rdquo; XXL https://amzn.to/3aogqOi Esbit https://amzn.to/3tYo1ux COMAIR 7001255 Pads (Pack of 500) 100% Cotton https://amzn.to/3jOzA2S Plastic bottle 250ml, empty, for refill-sets (10 pcs.) https://amzn.to/2Zmvzcg Bushbox LF Titanium https://amzn.to/3tUhpxc Bushbox Ultralight Outdoor Pocket Stove https://amzn.to/3dggL7u Trangia alcohol burner for storm cookers https://amzn.to/3b4TrXz Stabilotherm Hunter Pan with Folding Handle Open https://amzn.to/2LTAnmD Tatonka Multi Set Kochgeschirr with Alcohol Burner Tools and Knifes\nhttps://amzn.to/2PFzpfo Bison hunter\u0026rsquo;s ax 1879 500g https://amzn.to/3rXbJ3S Hunting Knife with Satin Blade https://amzn.to/2NqBWZI Böker Plus Outdoorsman Knife https://amzn.to/2OANiut Work Sharp 09DX005 Knife and Tool Sharpener, Ken Onion Edition https://amzn.to/3b3jxtY Work Sharp Ken Onion Edition Sharpening Band Assortment - 3/4 Inch x 12 Inch https://amzn.to/3qrzdOi Diamond / Ceramic Sharpener CD4 https://amzn.to/3befprk Sharpening stone, BearMoo 2-in-1 sharpening stone, sharpening stone for knives, grit 1000/4000, with non-slip silicone holder https://amzn.to/3rUHRoH Folding Mini Hand Shovel, Ultra-Compact https://amzn.to/2LRqiXb Semptec Urban survival technology spade https://amzn.to/3u3H1HZ The Friendly Swede Folding Saw Outdoor with Paracord https://amzn.to/3s92fTl Kombat UK Unisex Paracord Roll (non 550) https://amzn.to/3bbE3sv WINGONEER 550 Paracord Mil Spec Type III 7 Beach Parachute Cord 30.5 m https://amzn.to/2Zgq7b9 Olight Warrior Mini EDC LED Torch 1500 Lumen 190 Range IPX8 Rechargeable 5 Modes Tactical Torch for Outdoors https://amzn.to/37gxWC4 12 Premium Power Glow Sticks Knixs – Red Intensive https://amzn.to/3tTKgBd Wood carving tool set https://amzn.to/3d3PwML Manual multi-purpose wood drill bit, Scotch Eye Video Equipment - outdoor recordings\nhttps://amzn.to/3jU1RVQ DJI RS Axis Stabiliser Gimbal for Mirrorless DSLR Cameras https://amzn.to/3u02tO2 Rode MINIFUR-LAV Synthetic Fur Windshield for Microphones https://amzn.to/3dhhR2S Rode VideoMicro Compact On-Camera Microphone https://amzn.to/2ZjLl88 DJI Osmo Mobile 3 Smartphone Stabiliser https://amzn.to/3k101SP Sony Alpha 6600 E-Mount System Camera https://amzn.to/2N8xjDR Sony SEL-1655G Wide Angle Zoom Lens (16–55 mm, F2.8) ","date":"18 February 2021","externalUrl":null,"permalink":"/my-outdoor-equipment/","section":"My outdoor equipment","summary":"I am listing some of the outd oor equipment I use for my outdoor adventures on this page. Let me know what your experience is or what you are using.\n","title":"My outdoor equipment","type":"my-outdoor-equipment"},{"content":" The four factors that are working against us # Software development is more and more dependent on Dependencies and the frequency of deployments is increasing. Both trends together are pushing themselves higher. Another element that turns the delivery of software into a network bottleneck is the usage of compounded artefacts. And the last trend that is working against us, is the exploding amount of edges or better-called edge nodes.All four trends together are a challenge for the infrastructure.But what we could do against it?\nhttps://youtu.be/VSLZ4Q6ELEk\nEdge-Computing # Before we look at the acceleration strategies I will explain a bit the term \u0026ldquo;Edge\u0026rdquo; or better \u0026ldquo;Edge-Computing\u0026rdquo; because this is often used in this context.\nWhat is Edge or better edge computing?\nThe principle of edge computing states that data processing takes place at the Edge of the network. Which device is ultimately responsible for processing the data can differ depending on the application and the implementation of the concept.\nAn edge device is a device on the network periphery that generates, processes or forwards data itself. Examples of edge devices are smartphones, autonomous vehicles, sensors or IoT devices such as fire alarms.\nAn edge gateway is installed between the edge device and the network. It receives data from edge devices that do not have to be processed in real-time, processes specific data locally or selectively, sends the data to other services or central data centers. Edge gateways have wireless or wired interfaces to the edge devices and the communication networks for private or public clouds.\nPros of Edge Computing\nThe data processing takes place in the vicinity of the data source, minimising transmission and response times. Communication is possible almost in real-time. Simultaneously, the data throughput and the bandwidth usage reduction in the network, since only specific data that are not to be processed locally need to be transmitted to central data centres. Many functions can also be maintained even if the network or parts of the network fail—the performance of edge computing scales by providing more intelligent devices at the network periphery.\nCons of Edge Computing\nEdge computing offers more security due to the locally limited data storage, but this is only the case if appropriate security concepts are available for the decentralised devices, due to the heterogeneity and many different devices, the effort involved in implementing the security concepts increases.\nFog Computing\nEdge computing and fog computing are both decentralised data processing concepts. Fog Computing inserts another layer with the so-called Fog Nodes between the edge devices and the cloud. These are small, local data centres in the access areas of the cloud. These fog nodes collect the data from the edge devices. You select the data to be processed locally or decentrally and forward it to central servers or process it directly yourself.\nSelecting the best of both worlds means we are combining both principles of Edge- and Fog-Computing.\nWhat are the acceleration options for SW Distribution? # There are different strategies to scale the distribution of binaries, and every solution suits a specific use-case. We will not have a few on cloud solutions only because companies are operating worldwide and have to deal with different governmental regulations and restrictions. Additionally, to these restrictions, I want to highlight the need for hybrid solutions as well. Hybrid solutions are including on-prem resources as well as air gaped infrastructure used for high-security environments.\na) Custom Solution based on replication or scaling servers\nOne possibility to scale inside your network/architecture is scaling hardware and working with direct replication. Implementing this by yourself will most-likely consume a higher budget of workforce, knowledge, time and money based on the fact that this is not a trivial project. At the same time, this approach is bound into the borders of the infrastructure you have access to.\nb) P2P Networks\nPeer to Peer networks is based on equal nodes that are sharing the binaries between all nodes.The peer to peer approach implies that you will have a bunch of copies of your files. If you are downloading a file from the network, all nodes can serve parts independently. This approach of splitting up files and delivering from different nodes simultaneously to the requesting node leads to constant and efficient network usage and reduced download times.\nc) CDN - Content Delivery Network\nCDN\u0026rsquo;s are optimised to deliver large files over regions. The network itself is build-out of a huge number of nodes that are caching files for regional delivery. With this strategy, the original server will not be overloaded.\n_Check out on my**Youtube** Channel the video with the title \u0026#34;**DevSecOps - the Low hanging fruits** \u0026#34;.This video describes the balance between writing the code itself or adding a dependency in each Cloud-Native App layer. The question is, what does this mean for DevSecOps?_ JFrog Solution # With the three mentioned techniques you can build up huge and powerful architecture that fit´s to your needs. But the integration of all these technologies and implementing products is not easy. We faced this challenge as well and over the years we found solutions that we integrated into a single DevSecOps Platform called \u0026ldquo;The JFrog Platform \u0026ldquo;. I don´t want to give an overview of all components, for this check out my youtube channel. Here I want to focus on the components that are responsible for the Distribution of the binaries only.\nJFrog Distribution\nWith the JFrog Distribution, the knowledge about the content of the repositories and the corresponding metadata is used to provide a replication strategy. The replication solution is designed for internal and external repositories to bring the binaries all the way down to the place where it is needed. The infrastructure can be built in a hybrid model, including on-prem and cloud nodes.Even air-gapped solutions are possible with import/export mechanisms. In this scenario, we are focussing on a scalable caching mechanism that is optimised for reads.\nWhat is a Release Bundle?\nA Release bundle is a composition of binaries. These binaries can be of different types, like maven, Debian or Docker. The Release Bundle can be seen as a Bills Of Materials (BOM).The content and well as the Release Bundles itself are immutable. This immutability makes it possible to implement efficient caching and replication mechanisms across different networks and regions.\nWhat is an Edge Node in this context?\nAn Edge Node in our context is a node that will provide the functionality of a read-only Artifactory.With this Edge Node, the delivery process is optimised, and we will see that replication is done in a transactional way. The difference to the original meaning of an Edge Node is that this instance is not the consuming or producing element. This can be seen as a Fog-Node, that is the first layer above the real edge nodes layer.\nP2P Download\nThe P2P solution focuses on environments that need to handle download bursts inside the same network or region.This download bursts could be scenarios like \u0026ldquo;updating a server farm\u0026rdquo; or \u0026ldquo;updating a Microservice Mesh\u0026rdquo;. The usage is unidirectional, which means that the consumer is not updating from their side. They are just waiting for a new version and all consumer updating at the same time.This behaviour is a perfect case for the P2P solution. Artifactory, or an Edge Node in the same network or region, is influencing an update of all P2P Nodes with a new version of a binary. The consumer itself will request the binary from the P2P node and not from the Artifactory instance anymore.The responsible Artifactory instance manages the P2P nodes, which leads to zero maintenance on the user side. Have in mind, that the RBAC is active at the P2P nodes as well.\nCDN Distribution\nThe CDN Solution is optimised to deliver binaries to different parts of the world. We have it in two flavours. One is for the public and mostly used to distribute SDK\u0026rsquo;s, Drivers or other free available binaries. The other flavour is focussing on the private distribution.Whatever solution you are using, the RBAC defined inside the Access Module is respected, including solutions with Authentication and Authorisation and unique links including Access Tokens.\nConclusion # Ok, it is time for the conclusion.What we discussed today;\nWith the increasing amount of dependencies, a higher frequency of deployments and the constantly growing number of applications and edge-nodes, we are facing scalability challenges.\nWe had a look at three ways you could go to increase your delivery speed.The discussed solution based on\na) JFrog Distribution helps you build up a strong replication strategy inside your hybrid infrastructure to speed up the development cycle.\nb) JFrog P2P that will allow you to handle massive download bursts inside a network or region. This solution fits tasks that need to distribute binaries to a high number of consumers concurrently during download bursts.\nc) JFrog CDN to deliver binaries worldwide into regional data centres to make the experience for the consumer as best as possible.\nAll this is bundled into the JFrog DevSecOps Platform.\nCheers Sven\n","date":"14 February 2021","externalUrl":null,"permalink":"/posts/a-challenge-of-the-software-distribution/","section":"Posts","summary":" The four factors that are working against us # Software development is more and more dependent on Dependencies and the frequency of deployments is increasing. Both trends together are pushing themselves higher. Another element that turns the delivery of software into a network bottleneck is the usage of compounded artefacts. And the last trend that is working against us, is the exploding amount of edges or better-called edge nodes.All four trends together are a challenge for the infrastructure.But what we could do against it?\n","title":"A Challenge of the Software Distribution","type":"posts"},{"content":" What do the effects the news of the last few months can have to do with risk management and the presumption of storage, and why is it an elementary component of DevSecOps?\nIf you want to see this Post as a video, check-out the following from me Youtube Channel\nhttps://youtu.be/nXxOXgRY5_s\nWhat Has Happened So Far # Again and again, changes have happened that set things in motion that were considered to have been set. In some cases, services or products have been freely available for many years, or the type of restriction has not changed. I am taking one of the last changes as an occasion to show the resulting behavior and to formulate solutions that help you deal with it.\nIn software development, repositories are one of the central elements that enable you to efficiently deal with the abundance of dependencies in a software environment. A wide variety of types and associated technologies have evolved over the decades. But there is a common approach mostly resulted in a global central authority that is seen as an essential reference.\n**I examined the topic of the repository from a generic point of view in a little more detail on youtube. **\nhttps://youtu.be/Wk-rlZ904to\nAs an example, I would like to briefly show what a minimal technology stack can look like today. Java is used for the application itself, the dependencies of which are defined using maven. For this, we need access to maven repositories. Debian repositories [Why Debian Repos are mission-critical..] used for the operating system on which the application is running. The components that then packaged into Docker images use Docker registries, and finally, the applications orchestrated in a composition of Docker images using Kubernetes. Here alone, we are dealing with four different repository types. At this point, I have left out the need for generic repositories to provide the required tools used within the DevSecOps pipeline.\nDockerHub And Its Dominance # The example that inspired me to write this article was DockerHub\u0026rsquo;s announcements. Access to this service was free, and there were no further restrictions on storage space and storage duration for freely available Docker images. This fact has led to a large number of open source projects using this repository for their purposes. Over the years a whole network of dependencies between these images has built up.\nDocker Hub was in the news recently for two reasons.\nStorage Restrictions # Previously, Docker images were stored indefinitely on Dockerhub. On the one hand, this meant that nobody cared about the storage space of the Docker images. On the other hand, pretty much everyone has been counting on it not to change again. Unfortunately, that has now changed. The retention period for inactive Docker images has been reduced to six months. What doesn\u0026rsquo;t sound particularly critical at first turns out to be quite uncomfortable in detail.\nDownload Throttling # Docker has limited the download rate to 100 pulls per six hours for anonymous users, and 200 pulls per six hours for free accounts. Number 200 sounds pretty bearable. However, it makes sense to take a more detailed look here. 200 inquiries / 6h are 200 inquiries / 360min. We\u0026rsquo;re talking about 0.55 queries/minute at a constant query rate. First, many systems do more than one build and therefore requests, every 2 minutes. Second, if the limit is reached, it can take more than half a business day to regain access. The latter is to be viewed as very critical. As a rule, limit values given per hour, which then only leads to a delay of a little less than an hour. Six hours is a different order of magnitude.\nMaven and MavenCentral # If you look at the different technologies, a similar monoculture emerges in the Maven area. Here is the maven-central a singular point operated by one company. A larger company bought this company. What does this mean for the future of this repository? I don\u0026rsquo;t know. However, it is not uncommon for costs to be optimized after a takeover by another company. A legitimate question arises here; What economic advantage does the operator of such a central, free-of-charge infrastructure have?\nJDKs # There have been so many structural changes here that I\u0026rsquo;m not even sure what the official name is. But there is one thing I observe with eagle eyes in projects. Different versions, platforms and providers of the JDKs result in a source of joy in LTS projects that should not be underestimated. Here, too, it is not guaranteed how long the providers will keep the respective builds of a JDK for a platform. What is planned today can be optimized tomorrow. Here, too, you should take a look at the JDKs that are not only used internally but also by customers. Who has all the installers for the JDKs in use in stock? Are these JDKs also used within your own CI route, or do you trust the availability of specific Docker images?\nModerate Independence # How can this be countered now? The answer is straightforward. You get everything you need just once and then save it in your systems. And so we are running against the efforts of the last few years. As in most other cases, moderate use of this approach is recommended. More important than ever is the sensible use of freely available resources. It can help if a stringent retention tactic is used. Not everything has to be kept indefinitely. Many elements that are held in the caches are no longer needed after a while. Sophisticated handling of repositories and the nesting of resources helps here. Unfortunately, I cannot go into too much detail here, but it can be noted in short form.\nThe structure of the respective repositories enables, on the one hand, to create concrete compositions and, on the other hand, to carry out very efficient maintenance. Sources must be kept in individual repositories and then merged using virtual repositories. This process can be built up so efficiently that it can even drastically reduce the number of build cycles.\nDevSecOps - Risk Minimization # There is another advantage in dealing with the subject of \u0026ldquo;independence\u0026rdquo;. Because all files that are kept in their structures can be analyzed with regard to vulnerabilities and compliance, now when these elements are in one place, in a repository manager, I have a central location where I can scan them. The result is a complete dependency graph that includes the dependencies of an application but also the associated workbench. That, in turn, is one of the critical statements when you turn to the topic of DevSecOps. Security is like quality! It\u0026rsquo;s not just a tool; it\u0026rsquo;s not just a person responsible for it. It is a philosophy that has to run through the entire value chain.\nHappy Coding,\nSven Ruppert\n","date":"12 February 2021","externalUrl":null,"permalink":"/posts/devsecops-be-independent-again/","section":"Posts","summary":" What do the effects the news of the last few months can have to do with risk management and the presumption of storage, and why is it an elementary component of DevSecOps?\n","title":"DevSecOps - Be Independent Again","type":"posts"},{"content":"Hello and welcome to my DevSecOps post. Here in Germany, it\u0026rsquo;s winter right now, and the forests are quiet. The snow slows down everything, and it\u0026rsquo;s a beautiful time to move undisturbed through the woods.\nHere you can pursue your thoughts, and I had to think about a subject that customers or participants at conferences ask me repeatedly.\nThe question is almost always:\nWhat are the quick wins or low hanging fruits if you want to deal more with the topic of security in software development?\nAnd I want to answer this question right now!\nFor the lazy ones, you can see it as youtube video as well\nhttps://www.youtube.com/embed/lNqADishl8w\nLet\u0026rsquo;s start with the definition of a phrase that often used in the business world.\nMake Or Buy # Even as a software developer, you will often hear this phrase during meetings with the company\u0026rsquo;s management and sales part.\nThe phrase is called; \u0026ldquo;Make or Buy \u0026ldquo;. Typically, we have to decide if we want to do something ourselves or spend money to buy the requested functionality. It could be less or more functionality or different so that we have to adjust ourself to use it in our context.\nBut as a software developer, we have to deal with the same question every day. I am talking about dependencies. Should we write the source code by ourselves or just adding the next dependencies? Who will be responsible for removing bugs, and what is the total cost of this decision? But first, let\u0026rsquo;s take a look at the make-or-buy association inside the full tech-stack.\nDiff between Make / Buy on all layers. # If we are looking at all layers of a cloud-native stack to compare the value of \u0026ldquo;make\u0026rdquo; to \u0026ldquo;buy\u0026rdquo; we will see that the component \u0026ldquo;buy\u0026rdquo; is in all layers the bigger one. But first things first.\nThe first step is the development of the application itself.\nAssuming that we are working with Java and using maven as a dependency manager, we are most likely adding more lines of code indirectly as dependency compared to the number of lines we are writing by ourselves. The dependencies are the more prominent part, and third parties develop them. We have to be carefully, and it is good advice to check these external binaries for known vulnerabilities.\nWe should have the same behaviour regarding compliance and license usage. The next layer will be the operating system, in our case Linux.\nAnd again, we are adding some configuration files and the rest are existing binaries.\nThe result is an application running inside the operating system that is a composition of external binaries based on our configuration.\nThe two following layers, Docker and Kubernetes, are leading us to the same result. Until now, we are not looking at the tool-stack for the production line itself.\nAll programs and utilities that are directly or indirectly used under the hood called DevSecOps are some dependencies.\nAll layers\u0026rsquo; dependencies are the most significant part by far.\nChecking these binaries against known Vulnerabilities is the first logical step.\none time and recurring efforts for Compliance/Vulnerabilities # Comparing the effort of scanning against known Vulnerabilities and for Compliance Issues, we see a few differences.\nLet\u0026rsquo;s start with the Compliance issues.\nCompliance issues:\nThe first step will be defining what licenses are allowed at what part of the production line. This definition of allowed license includes the dependencies during the coding time and the usage of tools and runtime environments. Defining the non-critical license types should be checked by a specialised lawyer. With this list of white labelled license types, we can start using the machine to scan on a regular base the full tool stack. After the machine found a violation, we have to remove this element, and it must be replaced by another that is licensed under a white-labelled one.\nVulnerabilities:\nThe recurrent effort on this site is low compared to the amount of work that vulnerabilities are producing. A slightly different workflow is needed for the handling of found vulnerabilities. Without more significant preparations, the machine can do the work on a regular base as well. The identification of a vulnerability will trigger the workflow that includes human interaction. The vulnerability must be classified internally that leads to the decisions what the following action will be.\nCompliance Issues: just singular points in your full-stack # There is one other difference between Compliance Issues and Vulnerabilities. If there is a compliance issue, it is a singular point inside the overall environment. Just this single part is a defect and is not influencing other elements of the environment.\nVulnerabilities: can be combined into different attack vectors. # Vulnerabilities are a bit different. They do not only exist at the point where they are located. Additionally, they can be combined with other existing vulnerabilities in any additional layer of the environment. Vulnerabilities can be combined into different attack vectors. Every possible attack vector itself must be seen and evaluated. A set of minor vulnerabilities in different layers of the application can be combined into a highly critical risk.\nVulnerabilities: timeline from found until active in the production # I want to have an eye on as next is the timeline from vulnerability is found until the fix is in production. After a vulnerability is existing in a binary, we have nearly no control over the time until this is found. It depends on the person itself if the vulnerability is reported to the creator of the binary, a commercial security service, a government or it will be sold on a darknet marketplace. But, assuming that the information is reported to the binary creator itself, it will take some time until the data is publicly available. We have no control over the duration from finding the vulnerability to the time that the information is publicly available. The next period is based on the commercial aspect of this issue.\nAs a consumer, we can only get the information as soon as possible is spending money.\nThis state of affairs is not nice, but mostly the truth.\nNevertheless, at some point, the information is consumable for us. If you are using JFrog Xray , from the free tier, for example, you will get the information very fast. JFrog is consuming different security information resources and merging all information into a single vulnerability database. After this database is fed with new information, all JFrog Xray instances are updated. After this stage is reached, you can act.\nTest-Coverage is your safety-belt; try Mutation Testing. # Until now, the only thing you can do to speed up the information flow is spending money for professional security information aggregator. But as soon as the information is consumable for you, the timer runs. It depends on your environment how fast this security fix will be up and running in production. To minimise the amount of time a full automated CI Pipeline ist one of the critical factors.\nBut even more critical is excellent and robust test coverage.\nGood test coverage will allow you, to switch dependency versions immediately and push this change after a green test run into production. I recommend using a more substantial test coverage as pure line-coverages. The technique called \u0026ldquo;mutation test coverage\u0026rdquo; is a powerful one.\nMutation Test Coverage\nIf you want to know more about this on, check out my YouTube channel. I have a video that explains the theoretical part and the practical one for Java and Kotlin.\nhttps://www.youtube.com/embed/6Vej7YEOF8g\nThe need for a single point that understands all repo-types\nTo get a picture of the full impact graph based on all known vulnerabilities, it is crucial to understand all package managers included by the dependencies. Focussing on just one layer in the tech-stack is by far not enough.\nJFrog Artifactory provides information, including the vendor-specific metadata that is part of the package managers.\nJFrog Xray can consume all this knowledge and can scan all binaries that are hosted inside the repositories that are managed by Artifactory.\nVulnerabilities - IDE plugin # Shift Left means that Vulnerabilities must be eliminated as early as possible inside the production pipeline. One early-stage after the concept phase is the coding itself. At the moment you start adding dependencies to your project you are possibly adding Vulnerabilities as well.\nThe fastest way to get feedback regarding your dependencies is the JFrog IDE Plugin. This plugin will connect your IDE to your JFrog Xray Instance. The free tier will give you access to Vulnerability scanning. The Plugin is OpenSource and available for IntelliJ, VS-Code, Eclipse,\u0026hellip; If you need some additional features, make a feature request on GitHub or fork the Repository add your changes and make a merge request.\nTry it out by yourself -JFrog Free Tier\nHow to use the IDE plugin? # If you add a dependency to your project, the IDE Plugin can understand this information based on the used package manager. The IDE Plugin is connected to your JFrog Xray instance and will be queried if there is a change inside your project\u0026rsquo;s dependency definition. The information provided by Xray includes the known vulnerabilities of the added dependency. If there is a fixed version of the dependency available, the new version number will be shown.\nIf you want to see the IDE Plugin in Action without registering\nfor a Free Tier, have a look at my youtube video.\nhttps://www.youtube.com/embed/PsghzAf-ODU\nConclusion # With the JFrog Free Tier, you have the tools in your hands to practice Shift Left and pushing it into your IDE.\nCreate repositories for all included technologies, use Artifactory as a proxy for your binaries and let Xray scan the full stack.\nWith this, you have a complete impact graph based on your full-stack and the pieces of information about known Vulnerabilities as early as possible inside your production line.\nYou don\u0026rsquo;t have to wait until your CI Pipeline starts complaining. This will save a lot of your time.\n","date":"28 January 2021","externalUrl":null,"permalink":"/posts/the-quick-wins-of-devsecops/","section":"Posts","summary":"Hello and welcome to my DevSecOps post. Here in Germany, it’s winter right now, and the forests are quiet. The snow slows down everything, and it’s a beautiful time to move undisturbed through the woods.\n","title":"The quick Wins of DevSecOps","type":"posts"},{"content":" Let\u0026rsquo;s get in touch # Transform your digital landscape with world-class software development and developer relations services # In today\u0026rsquo;s fast-paced digital world, the right software solutions and a strong developer community can catapult your business to new heights. This is where I come in – your dedicated partner in tackling the complexities of software development and fostering dynamic developer relationships.\nWhy should you choose our services?\nCustomized Software Solutions : Software development services are tailored to your unique business needs, combining innovation with efficiency. From ideation to deployment, I help ensure your software is functional and one step ahead of the competition.\nStrengthening developer engagement : A vibrant developer community is at the heart of technological advancement. With my many years of international developer relations experience, I can help you build, maintain, and engage a community around your products by attracting advocates and driving innovation.\nExpertise you can trust : With over 20 years of experience, I bring extensive knowledge of software development and developer relations. My holistic approach ensures that your software meets market needs and is supported by an enthusiastic developer community.\nEnd-to-End Support : The journey doesn\u0026rsquo;t end with release. The role is to provide ongoing support, community management and iterative improvements based on real-world feedback.\nFuture-proof your business : Staying ahead is paramount in a rapidly evolving digital landscape. All with cutting-edge technologies and strategies to ensure your software and developer community stays at the forefront of innovation.\nWhat sets me apart?\nPersonalized Approach : I believe in solutions that reflect your unique brand identity and business goals. The personalized approach means you receive services perfectly tailored to your vision.\nTransparent communication : I value precise, continuous communication in every project phase. Stay informed and ensure the final product is exactly what you envisioned.\nCommitment to Excellence : Quality is at the heart of everything we do. From code to community engagement strategies, the focus is on delivering excellence.\nLet\u0026rsquo;s build something extraordinary together.\nContact me today to discuss how we can turn your ideas into reality and elevate your digital experience beyond the ordinary.\nEmail me at sven.ruppert@gmail.com or follow me on:\n","date":"28 January 2021","externalUrl":null,"permalink":"/contact/","section":"Contact","summary":" Let’s get in touch # Transform your digital landscape with world-class software development and developer relations services # In today’s fast-paced digital world, the right software solutions and a strong developer community can catapult your business to new heights. This is where I come in – your dedicated partner in tackling the complexities of software development and fostering dynamic developer relationships.\n","title":"Contact","type":"contact"},{"content":"","date":"28 January 2021","externalUrl":null,"permalink":"/bushcrafting/","section":"Bushcrafting","summary":"","title":"Bushcrafting","type":"bushcrafting"},{"content":" Impressum # Angaben gemäß § 5 TMG\nSven Ruppert Otto-Hahn-Str. 25 38116 Braunschweig\nVertreten durch:\nSven Ruppert Kontakt: Telefon: 0151-59100028 E-Mail: sven.ruppert@gmail.com\nVerantwortlich für den Inhalt nach § 55 Abs. 2 RStV :\nSven Ruppert Otto-Hahn-Str. 25 38116 Braunschweig\nHaftungsausschluss:\nHaftung für Inhalte\nDie Inhalte unserer Seiten wurden mit größter Sorgfalt erstellt. Für die Richtigkeit, Vollständigkeit und Aktualität der Inhalte können wir jedoch keine Gewähr übernehmen. Als Diensteanbieter sind wir gemäß § 7 Abs.1 TMG für eigene Inhalte auf diesen Seiten nach den allgemeinen Gesetzen verantwortlich. Nach §§ 8 bis 10 TMG sind wir als Diensteanbieter jedoch nicht verpflichtet, übermittelte oder gespeicherte fremde Informationen zu überwachen oder nach Umständen zu forschen, die auf eine rechtswidrige Tätigkeit hinweisen. Verpflichtungen zur Entfernung oder Sperrung der Nutzung von Informationen nach den allgemeinen Gesetzen bleiben hiervon unberührt. Eine diesbezügliche Haftung ist jedoch erst ab dem Zeitpunkt der Kenntnis einer konkreten Rechtsverletzung möglich. Bei Bekanntwerden von entsprechenden Rechtsverletzungen werden wir diese Inhalte umgehend entfernen.\nHaftung für Links\nUnser Angebot enthält Links zu externen Webseiten Dritter, auf deren Inhalte wir keinen Einfluss haben. Deshalb können wir für diese fremden Inhalte auch keine Gewähr übernehmen. Für die Inhalte der verlinkten Seiten ist stets der jeweilige Anbieter oder Betreiber der Seiten verantwortlich. Die verlinkten Seiten wurden zum Zeitpunkt der Verlinkung auf mögliche Rechtsverstöße überprüft. Rechtswidrige Inhalte waren zum Zeitpunkt der Verlinkung nicht erkennbar. Eine permanente inhaltliche Kontrolle der verlinkten Seiten ist jedoch ohne konkrete Anhaltspunkte einer Rechtsverletzung nicht zumutbar. Bei Bekanntwerden von Rechtsverletzungen werden wir derartige Links umgehend entfernen.\nUrheberrecht\nDie durch die Seitenbetreiber erstellten Inhalte und Werke auf diesen Seiten unterliegen dem deutschen Urheberrecht. Die Vervielfältigung, Bearbeitung, Verbreitung und jede Art der Verwertung außerhalb der Grenzen des Urheberrechtes bedürfen der schriftlichen Zustimmung des jeweiligen Autors bzw. Erstellers. Downloads und Kopien dieser Seite sind nur für den privaten, nicht kommerziellen Gebrauch gestattet. Soweit die Inhalte auf dieser Seite nicht vom Betreiber erstellt wurden, werden die Urheberrechte Dritter beachtet. Insbesondere werden Inhalte Dritter als solche gekennzeichnet. Sollten Sie trotzdem auf eine Urheberrechtsverletzung aufmerksam werden, bitten wir um einen entsprechenden Hinweis. Bei Bekanntwerden von Rechtsverletzungen werden wir derartige Inhalte umgehend entfernen.\nDatenschutz\nDie Nutzung unserer Webseite ist in der Regel ohne Angabe personenbezogener Daten möglich. Soweit auf unseren Seiten personenbezogene Daten (beispielsweise Name, Anschrift oder eMail-Adressen) erhoben werden, erfolgt dies, soweit möglich, stets auf freiwilliger Basis. Diese Daten werden ohne Ihre ausdrückliche Zustimmung nicht an Dritte weitergegeben. Wir weisen darauf hin, dass die Datenübertragung im Internet (z.B. bei der Kommunikation per E-Mail) Sicherheitslücken aufweisen kann. Ein lückenloser Schutz der Daten vor dem Zugriff durch Dritte ist nicht möglich. Der Nutzung von im Rahmen der Impressumspflicht veröffentlichten Kontaktdaten durch Dritte zur Übersendung von nicht ausdrücklich angeforderter Werbung und Informationsmaterialien wird hiermit ausdrücklich widersprochen. Die Betreiber der Seiten behalten sich ausdrücklich rechtliche Schritte im Falle der unverlangten Zusendung von Werbeinformationen, etwa durch Spam-Mails, vor.\nGoogle Analytics\nDiese Website benutzt Google Analytics, einen Webanalysedienst der Google Inc. (\u0026lsquo;\u0026lsquo;Google\u0026rsquo;\u0026rsquo;). Google Analytics verwendet sog. \u0026lsquo;\u0026lsquo;Cookies\u0026rsquo;\u0026rsquo;, Textdateien, die auf Ihrem Computer gespeichert werden und die eine Analyse der Benutzung der Website durch Sie ermöglicht. Die durch den Cookie erzeugten Informationen über Ihre Benutzung dieser Website (einschließlich Ihrer IP-Adresse) wird an einen Server von Google in den USA übertragen und dort gespeichert. Google wird diese Informationen benutzen, um Ihre Nutzung der Website auszuwerten, um Reports über die Websiteaktivitäten für die Websitebetreiber zusammenzustellen und um weitere mit der Websitenutzung und der Internetnutzung verbundene Dienstleistungen zu erbringen. Auch wird Google diese Informationen gegebenenfalls an Dritte übertragen, sofern dies gesetzlich vorgeschrieben oder soweit Dritte diese Daten im Auftrag von Google verarbeiten. Google wird in keinem Fall Ihre IP-Adresse mit anderen Daten der Google in Verbindung bringen. Sie können die Installation der Cookies durch eine entsprechende Einstellung Ihrer Browser Software verhindern; wir weisen Sie jedoch darauf hin, dass Sie in diesem Fall gegebenenfalls nicht sämtliche Funktionen dieser Website voll umfänglich nutzen können. Durch die Nutzung dieser Website erklären Sie sich mit der Bearbeitung der über Sie erhobenen Daten durch Google in der zuvor beschriebenen Art und Weise und zu dem zuvor benannten Zweck einverstanden.\nGoogle AdSense\nDiese Website benutzt Google Adsense, einen Webanzeigendienst der Google Inc., USA (\u0026lsquo;\u0026lsquo;Google\u0026rsquo;\u0026rsquo;). Google Adsense verwendet sog. \u0026lsquo;\u0026lsquo;Cookies\u0026rsquo;\u0026rsquo; (Textdateien), die auf Ihrem Computer gespeichert werden und die eine Analyse der Benutzung der Website durch Sie ermöglicht. Google Adsense verwendet auch sog. \u0026lsquo;\u0026lsquo;Web Beacons\u0026rsquo;\u0026rsquo; (kleine unsichtbare Grafiken) zur Sammlung von Informationen. Durch die Verwendung des Web Beacons können einfache Aktionen wie der Besucherverkehr auf der Webseite aufgezeichnet und gesammelt werden. Die durch den Cookie und/oder Web Beacon erzeugten Informationen über Ihre Benutzung dieser Website (einschließlich Ihrer IP-Adresse) werden an einen Server von Google in den USA übertragen und dort gespeichert. Google wird diese Informationen benutzen, um Ihre Nutzung der Website im Hinblick auf die Anzeigen auszuwerten, um Reports über die Websiteaktivitäten und Anzeigen für die Websitebetreiber zusammenzustellen und um weitere mit der Websitenutzung und der Internetnutzung verbundene Dienstleistungen zu erbringen. Auch wird Google diese Informationen gegebenenfalls an Dritte übertragen, sofern dies gesetzlich vorgeschrieben oder soweit Dritte diese Daten im Auftrag von Google verarbeiten. Google wird in keinem Fall Ihre IP-Adresse mit anderen Daten der Google in Verbindung bringen. Das Speichern von Cookies auf Ihrer Festplatte und die Anzeige von Web Beacons können Sie verhindern, indem Sie in Ihren Browser-Einstellungen \u0026lsquo;\u0026lsquo;keine Cookies akzeptieren\u0026rsquo;\u0026rsquo; wählen (Im MS Internet-Explorer unter \u0026lsquo;\u0026lsquo;Extras \u0026gt; Internetoptionen \u0026gt; Datenschutz \u0026gt; Einstellung\u0026rsquo;\u0026rsquo;; im Firefox unter \u0026lsquo;\u0026lsquo;Extras \u0026gt; Einstellungen \u0026gt; Datenschutz \u0026gt; Cookies\u0026rsquo;\u0026rsquo;); wir weisen Sie jedoch darauf hin, dass Sie in diesem Fall gegebenenfalls nicht sämtliche Funktionen dieser Website voll umfänglich nutzen können. Durch die Nutzung dieser Website erklären Sie sich mit der Bearbeitung der über Sie erhobenen Daten durch Google in der zuvor beschriebenen Art und Weise und zu dem zuvor benannten Zweck einverstanden.\nImpressum vom Impressum Generator der Kanzlei Hasselbach, Bonn\n","date":"28 January 2021","externalUrl":null,"permalink":"/impressum/","section":"Impressum","summary":"Impressum # Angaben gemäß § 5 TMG\nSven Ruppert Otto-Hahn-Str. 25 38116 Braunschweig\nVertreten durch:\nSven Ruppert Kontakt: Telefon: 0151-59100028 E-Mail: sven.ruppert@gmail.com\n","title":"Impressum","type":"impressum"},{"content":" Hi, I\u0026rsquo;m Sven Ruppert # I\u0026rsquo;m a software developer, architect and developer advocate with 20+ years of experience in enterprise Java development. Based in Braunschweig, Germany.\nBy day, I work on Java and Cybersecurity topics. By night, I\u0026rsquo;m a passionate bushcrafter who loves spending time in the woods.\nWhat I do # Software Development \u0026amp; Architecture Building scalable, secure, enterprise-grade applications with a focus on clean code, testability and maintainability.\nSecurity Specialised in application security, DevSecOps, secure coding practices and vulnerability analysis (CVSS, CWE, EPSS).\nDeveloper Relations \u0026amp; Advocacy International speaker, author and trainer. Helping developers learn, grow and build better software.\nWorkshops \u0026amp; Training Offering hands-on workshops on Java Streams, JUnit5 testing and Functional Programming. See Workshops.\nTechnologies \u0026amp; Expertise # Languages: Java, Kotlin, Python Frameworks: Vaadin, Spring, EclipseStore Security: SAST/DAST, OWASP, DevSecOps, CWE/CVE analysis Build \u0026amp; DevOps: Maven, Gradle, JFrog, SLSA, Docker Testing: JUnit5, Mutation Testing, TDD Speaking \u0026amp; Community # I speak at international conferences and meetups on topics including Java internals, application security, DevSecOps and Vaadin.\nBushcrafting # When I\u0026rsquo;m not coding, I\u0026rsquo;m out in the woods practising bushcraft — building shelters, navigating by map and compass, carving tools and cooking over an open fire. See the Bushcrafting section of my blog for articles and videos.\nContact # Email: sven.ruppert@gmail.com LinkedIn: linkedin.com/in/svenruppert Twitter/X: @SvenRuppert Mastodon: @svenruppert@mastodon.social ","externalUrl":null,"permalink":"/about/","section":"About Me","summary":"Hi, I’m Sven Ruppert # I’m a software developer, architect and developer advocate with 20+ years of experience in enterprise Java development. Based in Braunschweig, Germany.\n","title":"About Me","type":"about"},{"content":" Programming Java since 1996. Developer Advocate, conference speaker, author and Arctic expedition trainer. Former Developer Advocate at JFrog and Vaadin. DevOps Institute Ambassador, JCP Member and Oracle Developer Champion. Current Position # Developer Advocate – Freelancer 07.2024 – present DACH region · svenruppert.com Bridging the gap between development communities and product teams by advocating for developer-friendly products, engaging in hands-on coding, building demos and contributing directly to development projects. International conference speaker (US/CA, EU, Australia, Singapore) Technical content creation — blogs, books, articles (print \u0026 online) Live coding sessions and hands-on workshops Open-source community building and developer relations POCs for potential corporate customers Professional Experience # Developer Advocate 01.2020 – 07.2024 JFrog Responsible for DACH, Australia/New Zealand and Singapore regions. POCs for potential corporate customers Blogs, books and articles — online and print media Conference speaker: US/CA, EU, AU/NZ, Singapore Topics: Core Java, Cybersecurity, Secure Coding Practices Developer Advocate 04.2017 – 12.2019 Vaadin Responsible for DACH, Australia/New Zealand and Singapore regions. POCs for potential corporate customers Blogs, books and articles — online and print media Conference speaker: US/CA, EU, AU/NZ, Singapore Topics: Core Java, Kotlin, Vaadin and server-side technologies Head of R\u0026amp;D 2015 – 2017 Macros reply GmbH Responsible for the development of the new product generation. Improved QM/QA and team performance. Technical focus on performance scalability and architecture refactoring. Core Java Kotlin Vaadin CDI JAX-RS Hazelcast MapDB Principal IT Consultant 2014 – 2015 codecentric AG, Munich Responsible for customer development team (15+ developers). Improved QM/QA and delivery performance. Focus on performance scalability and architecture refactoring. Core Java CDI JAX-RS Hazelcast Vaadin Oracle Cassandra Principal Architect / Technology Scout 2013 – 2014 SiteOS AG, Munich Responsible for Swing-to-JavaFX migration of a 15-year-old product with 1 million lines of code. Core Java and performance improvements on client and server. Core Java CDI Swing JavaFX Hibernate Hazelcast (Interim) COO/CTO 2012 – 2013 Jtel GmbH, Munich Responsible for refactoring a web-based call centre product. Core Java and performance improvements on client and server. Core Java CDI JavaEE JSF SQL Lead Architect / Project Manager – Freelancer 2007 – 2014 Europe and Asia Java projects: server-side, persistence layer and distributed computing EU research project on distributed search engines and data/text mining for German SMEs QM/QA lead for large German tourist booking portal — managing up to 50 developers in India Core Java Hadoop Lucene/Solr Neo4J Vaadin Hibernate Lead Software Engineer 2004 – 2007 LINEAS Informationssysteme GmbH, Brunswick Development of web-based tools for MAN to optimise production of coaches. Performance optimisation of rules and algorithms. Core Java Struts Tomcat Senior Software Engineer 2002 – 2004 Institute of Process Data Processing, Brunswick Software development in wavelet-based graphical data compression systems for the Mars space project BEAGLE II. Core Java C Wavelets Software Engineer 2000 – 2002 Institute of Process Data Processing, Siegen Development of a distributed database system for the Fraunhofer Institute — Internet and multimedia databases. Core Java C++ CORBA PostgreSQL Software Developer 1999 – 2000 Institute of Theoretical Electro-Technology, Siegen Hardware design in medical technology — measuring pain evoked potentials (EEG). Developer 1998 – 1999 Steib GmbH, Siegen Development of embedded sensor systems for use in the Brunsbüttel nuclear power plant. C Java Community \u0026amp; Side Projects # TestFX – Open-Source Project Lead 2014 – present github.com/TestFX/TestFX Testing framework for automated UI testing of JavaFX applications. Integrates with JUnit, supports cross-platform testing and provides a high-level API for simulating user interactions. Advisory Board 2022 – present MicroStream Software GmbH, Regensburg Advisory board member for MicroStream — high-performance Java object persistence, eliminating the need for ORM and SQL. Ambassador 2021 – present DevOps Institute Ambassador for the DevOps Institute — advancing the human elements of DevOps through education, certifications and community engagement. JCP Member 2016 – present Java Community Process Active member of the Java Community Process, participating in the development and revision of Java technology specifications. Head of Cloud Native Security 2015 – present DOAG (Deutsche Oracle-Anwendergruppe) Leading the Cloud Native Security special interest group within the German Oracle Users Group. Founder \u0026amp; Organiser – Kotlin User Group Munich 2015 – present 1,700\u0026#43; Members · meetup.com Founded and organise the Kotlin User Group Munich, one of Germany's largest Kotlin communities. Academic Experience # Assistant Professor 2011 – 2012 University Iserlohn Course on distributed and polyglot persistence. Core Java Neo4J Cassandra Hadoop Vaadin Assistant Professor 2007 – 2009 Rhein-Erft Akademie GmbH, Cologne Courses on distributed and polyglot persistence, Core Java and Design Patterns. Lecturer 2000 – 2002 Institute of Process Data Processing, Siegen Lectures and seminars on database systems and CORBA/middleware for distributed systems. Education # Information Technology Technical University Braunschweig, Germany Microsystem Electronics University Siegen, Germany Electrical Engineering University of Applied Sciences Aachen, Germany Electronics Technician (IHK) Vocational High School, Iserlohn Three-year formal training in electronics and information technology, certified by the German Chamber of Commerce and Industry (IHK). Military Service German Air Force – Crisis Reaction Forces (KRK) Skills # Languages \u0026amp; Runtimes Skill Description Java Primary language since 1996. Expert-level knowledge from Java 1 through Java 24 — Virtual Threads, Records, Pattern Matching, Streams, Generics, bytecode instrumentation. Kotlin JVM-native development, coroutines, interoperability with Java ecosystems, Kotlin DSLs. Python Scripting, tooling, data processing and automation workflows. C / C++ Embedded systems, sensor programming, low-level hardware interfaces (nuclear plant sensors, EEG hardware, Mars project). SQL Relational database design, query optimisation, stored procedures across Oracle, PostgreSQL and DB2. Bash Shell scripting for build pipelines, deployment automation and server administration. Frameworks \u0026amp; Libraries Skill Description Vaadin Flow Full-stack Java web UI framework. Building server-side web applications with a pure Java API — no JavaScript required. Developer Advocate at Vaadin 2017–2019. EclipseStore / MicroStream High-performance Java object graph persistence. Eliminates ORM and SQL overhead by directly persisting Java objects. Advisory Board member at MicroStream. Spring Boot Rapid application development on the Spring ecosystem — REST APIs, security, data access, microservices. CDI Contexts and Dependency Injection for JavaEE/Jakarta EE — used extensively in enterprise projects. JAX-RS RESTful web services in Java EE environments. JUnit5 Advanced testing with parameterised tests, dynamic tests, custom extensions and TestFactory. Author of multiple articles and a workshop on effective JUnit5 usage. TestFX Open-source project lead. Automated UI testing framework for JavaFX applications — integrates with JUnit and supports cross-platform testing. Hazelcast In-memory data grid for distributed caching, clustering and computation across multiple Java enterprise projects. Security Skill Description OWASP Top 10 Deep knowledge of the ten most critical web application security risks — injection, XSS, broken access control, etc. — with practical mitigation strategies in Java. DevSecOps Integrating security practices throughout the development lifecycle: security as code, automated checks in CI/CD pipelines, shift-left security. SAST / DAST Static Application Security Testing (code analysis) and Dynamic Application Security Testing (runtime analysis). Tool selection, pipeline integration and result triage. CWE / CVE Common Weakness Enumeration and Common Vulnerabilities and Exposures — author of a dedicated blog series covering CWE-22, CWE-377, CWE-416, CWE-787, CWE-778 and more. CVSS Common Vulnerability Scoring System — calculating and interpreting severity scores for vulnerability prioritisation. EPSS Exploit Prediction Scoring System — probabilistic model for assessing the likelihood of vulnerability exploitation. Supply Chain Security Dependency confusion, cache poisoning, SBOM, artefact signing and integrity verification in Maven/Gradle build pipelines. SLSA Supply chain Levels for Software Artefacts — Linux Foundation framework for improving software supply chain integrity. Build \u0026amp; DevOps Skill Description Maven Expert-level: multi-module projects, custom plugins, dependency management, reproducible builds and security hardening of the Maven build pipeline. Gradle Build automation for Android and JVM projects. Custom tasks, build scans and incremental builds. JFrog Artifactory Universal artefact repository management. Binary management, access control and replication. Developer Advocate at JFrog 2020–2024. JFrog Xray Continuous security and compliance scanning of artefacts and dependencies — CVE detection, licence compliance and policy enforcement. Docker Containerisation of Java applications, multi-stage builds, image optimisation and container security scanning. Atlassian Stack Jira, Confluence, Bitbucket — project management, documentation and code review workflows for distributed teams. Databases Skill Description PostgreSQL Open-source relational database — schema design, query optimisation, indexing strategies and JDBC integration. Oracle Enterprise relational database — used in multiple large-scale enterprise projects (codecentric, SiteOS). Neo4J Graph database for relationship-heavy data models. Used in EU research project on distributed search engines and in academic courses. Cassandra Wide-column NoSQL database for high-availability and high-throughput distributed applications. Solr / Lucene Full-text search engine. Used in EU research project on distributed search engines and data/text mining for German SMEs. Hibernate ORM framework for Java — entity mapping, HQL, caching strategies and performance tuning. Publications \u0026amp; Speaking # Books Amazon Author Page Print Magazines JavaSpektrum, JavaAktuell, JavaPro DZone Core Member Sigs.de Author Profile Vaadin Blog Articles Entwickler.de Profile Informatik-Aktuell Profile Conferences US/CA · EU · Australia/NZ · Singapore ~200,000 IT magazine copies per year featuring articles in the German-speaking market.\nLanguages # Language Level German Native English Fluent (professional) Contact # sven.ruppert@gmail.com LinkedIn @SvenRuppert ","externalUrl":null,"permalink":"/cv/","section":"Curriculum Vitae","summary":" Programming Java since 1996. Developer Advocate, conference speaker, author and Arctic expedition trainer. Former Developer Advocate at JFrog and Vaadin. DevOps Institute Ambassador, JCP Member and Oracle Developer Champion. Current Position # Developer Advocate – Freelancer 07.2024 – present DACH region · svenruppert.com Bridging the gap between development communities and product teams by advocating for developer-friendly products, engaging in hands-on coding, building demos and contributing directly to development projects. International conference speaker (US/CA, EU, Australia, Singapore) Technical content creation — blogs, books, articles (print \u0026 online) Live coding sessions and hands-on workshops Open-source community building and developer relations POCs for potential corporate customers Professional Experience # Developer Advocate 01.2020 – 07.2024 JFrog Responsible for DACH, Australia/New Zealand and Singapore regions. POCs for potential corporate customers Blogs, books and articles — online and print media Conference speaker: US/CA, EU, AU/NZ, Singapore Topics: Core Java, Cybersecurity, Secure Coding Practices Developer Advocate 04.2017 – 12.2019 Vaadin Responsible for DACH, Australia/New Zealand and Singapore regions. POCs for potential corporate customers Blogs, books and articles — online and print media Conference speaker: US/CA, EU, AU/NZ, Singapore Topics: Core Java, Kotlin, Vaadin and server-side technologies Head of R\u0026D 2015 – 2017 Macros reply GmbH Responsible for the development of the new product generation. Improved QM/QA and team performance. Technical focus on performance scalability and architecture refactoring. Core Java Kotlin Vaadin CDI JAX-RS Hazelcast MapDB Principal IT Consultant 2014 – 2015 codecentric AG, Munich Responsible for customer development team (15+ developers). Improved QM/QA and delivery performance. Focus on performance scalability and architecture refactoring. Core Java CDI JAX-RS Hazelcast Vaadin Oracle Cassandra Principal Architect / Technology Scout 2013 – 2014 SiteOS AG, Munich Responsible for Swing-to-JavaFX migration of a 15-year-old product with 1 million lines of code. Core Java and performance improvements on client and server. Core Java CDI Swing JavaFX Hibernate Hazelcast (Interim) COO/CTO 2012 – 2013 Jtel GmbH, Munich Responsible for refactoring a web-based call centre product. Core Java and performance improvements on client and server. Core Java CDI JavaEE JSF SQL Lead Architect / Project Manager – Freelancer 2007 – 2014 Europe and Asia Java projects: server-side, persistence layer and distributed computing EU research project on distributed search engines and data/text mining for German SMEs QM/QA lead for large German tourist booking portal — managing up to 50 developers in India Core Java Hadoop Lucene/Solr Neo4J Vaadin Hibernate Lead Software Engineer 2004 – 2007 LINEAS Informationssysteme GmbH, Brunswick Development of web-based tools for MAN to optimise production of coaches. Performance optimisation of rules and algorithms. Core Java Struts Tomcat Senior Software Engineer 2002 – 2004 Institute of Process Data Processing, Brunswick Software development in wavelet-based graphical data compression systems for the Mars space project BEAGLE II. Core Java C Wavelets Software Engineer 2000 – 2002 Institute of Process Data Processing, Siegen Development of a distributed database system for the Fraunhofer Institute — Internet and multimedia databases. Core Java C++ CORBA PostgreSQL Software Developer 1999 – 2000 Institute of Theoretical Electro-Technology, Siegen Hardware design in medical technology — measuring pain evoked potentials (EEG). Developer 1998 – 1999 Steib GmbH, Siegen Development of embedded sensor systems for use in the Brunsbüttel nuclear power plant. C Java Community \u0026 Side Projects # TestFX – Open-Source Project Lead 2014 – present github.com/TestFX/TestFX Testing framework for automated UI testing of JavaFX applications. Integrates with JUnit, supports cross-platform testing and provides a high-level API for simulating user interactions. Advisory Board 2022 – present MicroStream Software GmbH, Regensburg Advisory board member for MicroStream — high-performance Java object persistence, eliminating the need for ORM and SQL. Ambassador 2021 – present DevOps Institute Ambassador for the DevOps Institute — advancing the human elements of DevOps through education, certifications and community engagement. JCP Member 2016 – present Java Community Process Active member of the Java Community Process, participating in the development and revision of Java technology specifications. Head of Cloud Native Security 2015 – present DOAG (Deutsche Oracle-Anwendergruppe) Leading the Cloud Native Security special interest group within the German Oracle Users Group. Founder \u0026 Organiser – Kotlin User Group Munich 2015 – present 1,700+ Members · meetup.com Founded and organise the Kotlin User Group Munich, one of Germany's largest Kotlin communities. Academic Experience # Assistant Professor 2011 – 2012 University Iserlohn Course on distributed and polyglot persistence. Core Java Neo4J Cassandra Hadoop Vaadin Assistant Professor 2007 – 2009 Rhein-Erft Akademie GmbH, Cologne Courses on distributed and polyglot persistence, Core Java and Design Patterns. Lecturer 2000 – 2002 Institute of Process Data Processing, Siegen Lectures and seminars on database systems and CORBA/middleware for distributed systems. Education # Information Technology Technical University Braunschweig, Germany Microsystem Electronics University Siegen, Germany Electrical Engineering University of Applied Sciences Aachen, Germany Electronics Technician (IHK) Vocational High School, Iserlohn Three-year formal training in electronics and information technology, certified by the German Chamber of Commerce and Industry (IHK). Military Service German Air Force – Crisis Reaction Forces (KRK) Skills # Languages \u0026 Runtimes Skill Description Java Primary language since 1996. Expert-level knowledge from Java 1 through Java 24 — Virtual Threads, Records, Pattern Matching, Streams, Generics, bytecode instrumentation. Kotlin JVM-native development, coroutines, interoperability with Java ecosystems, Kotlin DSLs. Python Scripting, tooling, data processing and automation workflows. C / C++ Embedded systems, sensor programming, low-level hardware interfaces (nuclear plant sensors, EEG hardware, Mars project). SQL Relational database design, query optimisation, stored procedures across Oracle, PostgreSQL and DB2. Bash Shell scripting for build pipelines, deployment automation and server administration. Frameworks \u0026 Libraries Skill Description Vaadin Flow Full-stack Java web UI framework. Building server-side web applications with a pure Java API — no JavaScript required. Developer Advocate at Vaadin 2017–2019. EclipseStore / MicroStream High-performance Java object graph persistence. Eliminates ORM and SQL overhead by directly persisting Java objects. Advisory Board member at MicroStream. Spring Boot Rapid application development on the Spring ecosystem — REST APIs, security, data access, microservices. CDI Contexts and Dependency Injection for JavaEE/Jakarta EE — used extensively in enterprise projects. JAX-RS RESTful web services in Java EE environments. JUnit5 Advanced testing with parameterised tests, dynamic tests, custom extensions and TestFactory. Author of multiple articles and a workshop on effective JUnit5 usage. TestFX Open-source project lead. Automated UI testing framework for JavaFX applications — integrates with JUnit and supports cross-platform testing. Hazelcast In-memory data grid for distributed caching, clustering and computation across multiple Java enterprise projects. Security Skill Description OWASP Top 10 Deep knowledge of the ten most critical web application security risks — injection, XSS, broken access control, etc. — with practical mitigation strategies in Java. DevSecOps Integrating security practices throughout the development lifecycle: security as code, automated checks in CI/CD pipelines, shift-left security. SAST / DAST Static Application Security Testing (code analysis) and Dynamic Application Security Testing (runtime analysis). Tool selection, pipeline integration and result triage. CWE / CVE Common Weakness Enumeration and Common Vulnerabilities and Exposures — author of a dedicated blog series covering CWE-22, CWE-377, CWE-416, CWE-787, CWE-778 and more. CVSS Common Vulnerability Scoring System — calculating and interpreting severity scores for vulnerability prioritisation. EPSS Exploit Prediction Scoring System — probabilistic model for assessing the likelihood of vulnerability exploitation. Supply Chain Security Dependency confusion, cache poisoning, SBOM, artefact signing and integrity verification in Maven/Gradle build pipelines. SLSA Supply chain Levels for Software Artefacts — Linux Foundation framework for improving software supply chain integrity. Build \u0026 DevOps Skill Description Maven Expert-level: multi-module projects, custom plugins, dependency management, reproducible builds and security hardening of the Maven build pipeline. Gradle Build automation for Android and JVM projects. Custom tasks, build scans and incremental builds. JFrog Artifactory Universal artefact repository management. Binary management, access control and replication. Developer Advocate at JFrog 2020–2024. JFrog Xray Continuous security and compliance scanning of artefacts and dependencies — CVE detection, licence compliance and policy enforcement. Docker Containerisation of Java applications, multi-stage builds, image optimisation and container security scanning. Atlassian Stack Jira, Confluence, Bitbucket — project management, documentation and code review workflows for distributed teams. Databases Skill Description PostgreSQL Open-source relational database — schema design, query optimisation, indexing strategies and JDBC integration. Oracle Enterprise relational database — used in multiple large-scale enterprise projects (codecentric, SiteOS). Neo4J Graph database for relationship-heavy data models. Used in EU research project on distributed search engines and in academic courses. Cassandra Wide-column NoSQL database for high-availability and high-throughput distributed applications. Solr / Lucene Full-text search engine. Used in EU research project on distributed search engines and data/text mining for German SMEs. Hibernate ORM framework for Java — entity mapping, HQL, caching strategies and performance tuning. Publications \u0026 Speaking # Books Amazon Author Page Print Magazines JavaSpektrum, JavaAktuell, JavaPro DZone Core Member Sigs.de Author Profile Vaadin Blog Articles Entwickler.de Profile Informatik-Aktuell Profile Conferences US/CA · EU · Australia/NZ · Singapore ~200,000 IT magazine copies per year featuring articles in the German-speaking market.\n","title":"Curriculum Vitae","type":"cv"},{"content":"A selection of projects I have built, contributed to or maintain.\nURL Shortener (Core Java) # A full URL shortener built step by step in Core Java — no frameworks, just the JDK. Documented in a 14-part Advent Calendar series on this blog.\nTech: Core Java · Vaadin Flow · Server-Sent Events · REST Series: Advent Calendar 2025\nTypeTool # An open-source project for type-safe generic programming in Java.\nTech: Java · Generics\nEclipseStore Demos # Hands-on demos showing how to use EclipseStore for high-performance Java object persistence — from basic storage to complex data structures and the high-performance serialiser.\nTech: Java · EclipseStore Articles: EclipseStore category\nVaadin Flow Examples # A growing collection of Vaadin Flow examples covering: file upload/download, REST integration, Observer pattern, i18n, component extraction, login, and more.\nTech: Java · Vaadin Flow · Core Java Articles: Vaadin category\nBushcrafting App # A companion app for bushcrafting and outdoor navigation — covering UTM/WGS84 coordinates, MilRad, magnetic anomalies and map reading.\nTech: Java · Navigation APIs Articles: Bushcrafting category\nTinkerForge IoT with Java # Integration of TinkerForge hardware sensors with Java for IoT prototyping.\nTech: Java · TinkerForge SDK Article: IoT with TinkerForge and Java\nWorkshops # See the Workshops page for structured training materials on Java Streams, JUnit5 and Functional Programming.\n","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"A selection of projects I have built, contributed to or maintain.\nURL Shortener (Core Java) # A full URL shortener built step by step in Core Java — no frameworks, just the JDK. Documented in a 14-part Advent Calendar series on this blog.\n","title":"Projects","type":"projects"},{"content":"","externalUrl":null,"permalink":"/search/","section":"Sven Ruppert","summary":"","title":"Search","type":"page"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"},{"content":"","externalUrl":null,"permalink":"/videos/","section":"Videos","summary":"","title":"Videos","type":"videos"}]