Search Results for author:
Found 78 papers, 39 papers with code
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
This work represents a crucial step toward establishing global standards for AI risk and reliability evaluation while acknowledging the need for continued development in areas such as multiturn interactions, multimodal understanding, coverage of additional languages, and emerging hazard categories.
An Efficient Model Maintenance Approach for MLOps
Thus, we propose an improved MLOps pipeline, a new model maintenance approach and a Similarity Based Model Reuse (SimReuse) tool to address the challenges of ML model maintenance.
Fault Localization in Deep Learning-based Software: A System-level Approach
Over the past decade, Deep Learning (DL) has become an integral part of our daily lives.
Towards Optimizing SQL Generation via LLM Routing
Text-to-SQL enables users to interact with databases through natural language, simplifying access to structured data.
In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators
In contrast, a model trained solely on real-world data achieved mAPs of 0. 8 and 0. 75 for use case 1 and use case 2 after more than 25 epochs.
Toward Debugging Deep Reinforcement Learning Programs with RLExplorer
In this paper, we propose RLExplorer, the first fault diagnosis approach for DRL-based software systems.
What Information Contributes to Log-based Anomaly Detection? Insights from a Configurable Transformer-Based Approach
In this work, we propose a configurable Transformer-based anomaly detection model that can capture the semantic, sequential, and temporal information in the log data and allows us to configure the different types of information as the model's features.
Trimming the Risk: Towards Reliable Continuous Training for Deep Learning Inspection Systems
Then, a fine-tuning of the original DL model is executed on the filtered inputs while validating on a mixture of recent production and original datasets.
TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models
The tasks' characteristics can be used to identify shortcomings within existing benchmarks.
DeepCodeProbe: Towards Understanding What Models Trained on Code Learn
Machine learning models trained on code and related artifacts offer valuable support for software maintenance but suffer from interpretability issues due to their complex internal variables.
Mining Action Rules for Defect Reduction Planning
In this paper, we introduce CounterACT, a Counterfactual ACTion rule mining approach that can generate defect reduction plans without black-box models.
Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs
In this study, we propose a self-refinement method aimed at improving the reliability of code generated by LLMs by minimizing the number of bugs before execution, without human intervention, and in the absence of test cases.
PathOCL: Path-Based Prompt Augmentation for OCL Generation with GPT-4
Moreover, the average prompt size crafted using PathOCL significantly decreases when scaling the size of the UML class models.
Introducing v0.5 of the AI Safety Benchmark from MLCommons
We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0. 5 benchmark.
Machine Learning Robustness: A Primer
This chapter explores the foundational concept of robustness in Machine Learning (ML) and its integral role in establishing trustworthiness in Artificial Intelligence (AI) systems.
Bugs in Large Language Models Generated Code: An Empirical Study
The bug patterns are presented in the form of a taxonomy.
Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Code
Therefore, auditing code developed using LLMs is challenging, as it is difficult to reliably assert if an LLM used during development has been trained on specific copyrighted codes, given that we do not have access to the training datasets of these models.
ChatGPT vs LLaMA: Impact, Reliability, and Challenges in Stack Overflow Discussions
Since its release in November 2022, ChatGPT has shaken up Stack Overflow, the premier platform for developers' queries on programming and software development.
Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends
Based on our qualitative analysis, we present a taxonomy of the challenges and benefits associated with PTM reuse within this community.
Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study
Practitioners and researchers can leverage our findings to improve deep learning bug reproducibility.
Refining GPT-3 Embeddings with a Siamese Structure for Technical Post Duplicate Detection
When applied to the dataset we constructed with a recent Stack Overflow dump, our approach attains a Top-1, Top-5, and Top-30 accuracy of 23. 1%, 43. 9%, and 68. 9%, respectively.
Characterizing and Classifying Developer Forum Posts with their Intentions
The modeling of the intentions of posts can provide an extra dimension to the current tag taxonomy.
An empirical study of testing machine learning in the wild
To fill this gap, we perform the first fine-grained empirical study on ML testing in the wild to identify the ML properties being tested, the testing strategies, and their implementation throughout the ML workflow.
GIST: Generated Inputs Sets Transferability in Deep Learning
Given a property selected by a user (e. g., neurons covered, faults), GIST enables the selection of good test sets from the point of view of this property among available test sets.
Detection and Evaluation of bias-inducing Features in Machine learning
In this study, we propose an approach for systematically identifying all bias-inducing features of a model to help support the decision-making of domain experts.
Data Cleaning and Machine Learning: A Systematic Literature Review
First, it aims to summarize the latest approaches for data cleaning for ML and ML for data cleaning.
Reinforcement learning informed evolutionary search for autonomous systems testing
Evolutionary search-based techniques are commonly used for testing autonomous robotic systems.
Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challenges
In this paper, we propose an empirical study on Stack Overflow (SO), the most popular Q&A forum for developers, to uncover and understand the challenges practitioners faced when deploying DRL systems.
An Intentional Forgetting-Driven Self-Healing Method For Deep Reinforcement Learning Systems
Dr. DRL successfully helps agents to adapt to 19. 63% of drifted environments left unsolved by vanilla CL while maintaining and even enhancing by up to 45% the obtained rewards for drifted environments that are resolved by both approaches.
On the Effectiveness of Log Representation for Log-based Anomaly Detection
We believe our comprehensive comparison of log representation techniques can help researchers and practitioners better understand the characteristics of different log representation techniques and provide them with guidance for selecting the most suitable ones for their ML-based log analysis workflow.
Bug Characterization in Machine Learning-based Systems
Based on our results, fixing ML bugs are more costly and ML components are more error-prone, compared to non-ML bugs and non-ML components respectively.
An Empirical Study on Bugs Inside PyTorch: A Replication Study
Software systems are increasingly relying on deep learning components, due to their remarkable capability of identifying complex data patterns and powering intelligent behaviour.
Quality Issues in Machine Learning Software Systems
We validate the identified quality issues via a survey with ML practitioners.
Responsible Design Patterns for Machine Learning Pipelines
In this paper, we propose a comprehensive framework incorporating RDPs into ML pipelines to mitigate risks and ensure the ethical development of AI systems.
Leveraging Data Mining Algorithms to Recommend Source Code Changes
Objectives: This paper proposes an automatic method for recommending source code changes using four data mining algorithms.
On Codex Prompt Engineering for OCL Generation: An Empirical Study
We investigate the reliability of OCL constraints generated by Codex from natural language specifications.
Mutation Testing of Deep Reinforcement Learning Based on Real Faults
This allows us to compare different mutation killing definitions based on existing approaches, as well as to analyze the behavior of the obtained mutation operators and their potential combinations called Higher Order Mutation(s) (HOM).
AmbieGen: A Search-based Framework for Autonomous Systems Testing
To address this challenge, we introduce AmbieGen, a search-based test case generation framework for autonomous systems.
Can Ensembling Pre-processing Algorithms Lead to Better Machine Learning Fairness?
As machine learning (ML) systems get adopted in more critical areas, it has become increasingly crucial to address the bias that could occur in these systems.
An Empirical Study of Library Usage and Dependency in Deep Learning Frameworks
In this paper, we present a qualitative and quantitative analysis of the most frequent dl libraries combination, the distribution of dl library dependencies across the ml workflow, and formulate a set of recommendations to (i) hardware builders for more optimized accelerators and (ii) library builder for more refined future releases.
Reliable Malware Analysis and Detection using Topology Data Analysis
Next, we compare the different TDA techniques (i. e., persistence homology, tomato, TDA Mapper) and existing techniques (i. e., PCA, UMAP, t-SNE) using different classifiers including random forest, decision tree, xgboost, and lightgbm.
Physics-Guided Adversarial Machine Learning for Aircraft Systems Simulation
In the context of aircraft system performance assessment, deep learning technologies allow to quickly infer models from experimental measurements, with less detailed system knowledge than usually required by physics-based modeling.
SmOOD: Smoothness-based Out-of-Distribution Detection Approach for Surrogate Neural Networks in Aircraft Design
Hybrid surrogate optimization maintains high results quality while providing rapid design assessments when both the surrogate model and the switch mechanism for eventually transitioning to the HF model are calibrated properly.
An Empirical Study on the Usage of Automated Machine Learning Tools
Machine learning (ML) practitioners use AutoML tools to automate and optimize the process of feature engineering, model training, and hyperparameter optimization and so on.
A Comparison of Reinforcement Learning Frameworks for Software Testing Tasks
In this paper, we empirically investigate the applications of carefully selected DRL algorithms on two important software testing tasks: test case prioritization in the context of Continuous Integration (CI) and game testing.
Quality issues in Machine Learning Software Systems
This empirical study aims to identify a catalog of bad-practices related to poor quality in MLSSs.
A Probabilistic Framework for Mutation Testing in Deep Neural Networks
Methods: In this work, we propose a Probabilistic Mutation Testing (PMT) approach that alleviates the inconsistency problem and allows for a more consistent decision on whether a mutant is killed or not.
DiverGet: A Search-Based Software Testing Approach for Deep Neural Network Quantization Assessment
Quantization is one of the most applied Deep Neural Network (DNN) compression strategies, when deploying a trained DNN model on an embedded system or a cell phone.
Dev2vec: Representing Domain Expertise of Developers in an Embedding Space
Moreover, our findings suggest that ``issue resolving history'' of developers is the most informative source of information to represent the domain expertise of developers in embedding spaces.
GitHub Copilot AI pair programmer: Asset or Liability?
In this paper, we study the capabilities of Copilot in two different programming tasks: (i) generating (and reproducing) correct and efficient solutions for fundamental algorithmic problems, and (ii) comparing Copilot's proposed solutions with those of human programmers on a set of programming tasks.
Threat Assessment in Machine Learning based Systems
Attacks from the AI Incident Database and the literature are used to identify vulnerabilities and new types of threats that were not documented in ATLAS.
An Empirical Study of Challenges in Converting Deep Learning Models
Usually DL models are developed and trained using DL frameworks that have their own internal mechanisms/formats to represent and train DL models, and usually those formats cannot be recognized by other frameworks.
Bugs in Machine Learning-based Systems: A Faultload Benchmark
Although most of these tools use bugs' lifecycle, there is no standard benchmark of bugs to assess their performance, compare them and discuss their advantages and weaknesses.
Never trust, always verify : a roadmap for Trustworthy AI?
In this paper, we examine trust in the context of AI-based systems to understand what it means for an AI system to be trustworthy and identify actions that need to be undertaken to ensure that AI systems are trustworthy.
Studying the Practices of Deploying Machine Learning Projects on Docker
Studies have recently explored the use of Docker for deploying general software projects with no specific focus on how Docker is used to deploy ML-based projects.
Fool SHAP with Stealthily Biased Sampling
SHAP explanations aim at identifying which features contribute the most to the difference in model prediction at a specific input versus a background distribution.
The Different Faces of AI Ethics Across the World: A Principle-Implementation Gap Analysis
Next, we analyze the current level of AI readiness and current implementations of ethical AI principles in different countries, to identify gaps in the implementation of AI principles and their causes.
Testing Feedforward Neural Networks Training Programs
All these model training steps can be error-prone.
A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems
We compared three configurations of AmbieGen: based on a single objective genetic algorithm, multi objective, and random search.
Machine Learning Application Development: Practitioners' Insights
In this paper, we report about a survey that aimed to understand the challenges and best practices of ML application development.
Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow
This paper presents the first empirical study of Keras and TensorFlow silent bugs, and their impact on users' programs.
On Assessing The Safety of Reinforcement Learning algorithms Using Formal Methods
The increasing adoption of Reinforcement Learning in safety-critical systems domains such as autonomous vehicles, health, and aviation raises the need for ensuring their safety.
An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets
We report the results of an empirical study that we conducted to determine the feasibility of developing an ensemble engine by combining the polarity labels of stand-alone SE-specific sentiment detectors.
Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set
In this work, instead of aiming at reducing the under-specification of model explanations, we fully embrace it and extract logical statements about feature attributions that are consistent across all models with good empirical performance (i. e. all models in the Rashomon Set).
The challenge of reproducible ML: an empirical study on the impact of bugs
This study attempts to quantify the impact that the occurrence of bugs in a popular ML framework, PyTorch, has on the performance of trained models.
The Forgotten Role of Search Queries in IR-based Bug Localization: An Empirical Study
We confirmed that the state-of-the-art query construction approaches are indeed not sufficient for constructing appropriate queries (for bug localization) from certain natural language-only bug reports although they contain such queries.
Models of Computational Profiles to Study the Likelihood of DNN Metamorphic Test Cases
We investigate the distribution of computational profile likelihood of metamorphic test cases with respect to the likelihood distributions of training, test and error control cases.
How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
Method: We conduct a Systematic Literature Review (SLR) of research papers published between 2015 to 2020, covering topics related to the certification of ML systems.
HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks
Deep Neural Networks (DNN) applications are increasingly becoming a part of our everyday life, from medical applications to autonomous cars.
Data Driven Testing of Cyber Physical Systems
The most common approach for pre-deployment testing is to model the system and run simulations with models or software in the loop.
Mining API Usage Scenarios from Stack Overflow
Each task consists of a code example, the task description, and the reactions of developers towards the code example.
Software Engineering
Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approach
To allow for the automatic detection of faults in DRL programs, we have defined a meta-model of DRL programs and developed DRLinter, a model-based fault detection approach that leverages static analysis and graph transformations.
SIGMA : Strengthening IDS with GAN and Metaheuristics Attacks
More specifically, we propose a new method named SIGMA, that leverages adversarial examples to strengthen IDS against new types of attacks.
Studying Software Engineering Patterns for Designing Machine Learning Systems
Machine-learning (ML) techniques have become popular in the recent years.
DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks
To overcome these limitations, we propose, DeepEvolution, a novel search-based approach for testing DL models that relies on metaheuristics to ensure a maximum diversity in generated test cases.
TFCheck : A TensorFlow Library for Detecting Training Issues in Neural Network Programs
In this paper, we examine training issues in ML programs and propose a catalog of verification routines that can be used to detect the identified issues, automatically.
Machine Learning Software Engineering in Practice: An Industrial Case Study
Especially, retail customers of SAP deal with millions of sales transactions for their day-to-day business.
A Machine-learning Based Ensemble Method For Anti-patterns Detection
In this paper, we present SMAD (SMart Aggregation of Anti-patterns Detectors), a machine-learning based ensemble method to aggregate various anti-patterns detection approaches on the basis of their internal detection rules.
