Roy Mwavita

×

🕷️ Automated Job Scraping & Dashboard

Project Type: Web Scraping & Interactive Dashboard

Status: 🟢 Live Demonstration Project

Tools & Technologies: R, rvest, tidyverse, lubridate, Shiny, shinydashboard, highcharter, DT, plotly, wordcloud, tm, waiter, shinyapps.io, GitHub

---

🧠 Problem Statement

Job seekers often spend hours manually refreshing job websites, scrolling through cluttered pages, and trying to identify new opportunities. Websites like MyJobMag don't always display vacancies in clear chronological order, and the interface is filled with advertisements that make navigation difficult. This results in wasted time, missed opportunities, and a frustrating job search experience.

---

🎯 Solution

I built an automated web scraping pipeline that extracts job postings from MyJobMag, processes them in R, and displays the results through an interactive Shiny dashboard.

The system scrapes job titles, companies, dates, and links from multiple pages, cleans and structures the data, filters jobs by date range and company, visualizes hiring trends, and provides a clean, ad-free interface for job searching.

---

⚙️ System Architecture

• 🌐 MyJobMag Website – Source of job postings
• 🕷️ rvest – Web scraping engine
• 🧹 R + tidyverse – Data cleaning and transformation
• 📅 lubridate – Date parsing and filtering
• 📊 Shiny Dashboard – Interactive visualization
• ☁️ shinyapps.io – Cloud deployment

---

🕷️ Data Collection Process

The scraper performs the following steps:

• Extracts job listings from MyJobMag pages
• Parses job titles to extract company names from " at " patterns
• Filters jobs based on recency (configurable date range)
• Removes duplicate entries using unique URLs
• Stores the data in a structured format for analysis

Data Collected:

• Job Title (cleaned)
• Company Name (extracted from title)
• Date Posted (parsed and validated)
• Job URL (complete link)
• Page Source (for pagination tracking)

---

📊 Dashboard Features

Key Metrics:

• Total jobs scraped
• New jobs today
• Number of hiring companies

Visualizations:

• 📈 Jobs posted over time (trend analysis)
• 🏆 Top hiring companies (bar chart)
• ☁️ Word cloud of job titles (keyword patterns)
• 📅 Jobs by day of week (best days to check)

Interactive Filters:

• Date range selector
• Company name filter
• Real-time data refresh

Data Export:

• Download filtered results as CSV

---

🔧 Technical Highlights

Scraping Logic:

  myjobmag_jobs <- function(base_url, max_pages = 7, days_back = 7) {
    # Automated scraping with pagination and error handling
    # Returns cleaned, deduplicated job data as a tibble
  }

Title Parsing:

  # Extract company from "Job Title at Company Name"
  if (str_detect(title, " at ")) {
    parts <- str_split(title, " at ", simplify = TRUE)
    job_title <- str_trim(parts[1])
    company <- str_trim(parts[2])
  }

---

📸 Dashboard Preview

Key Views:

• Overview Dashboard – KPI cards and trend charts
• Job Listings – Searchable, sortable data table
• Word Cloud – Visual analysis of job title keywords

---

🚀 Impact

• Eliminated manual scrolling – Jobs are automatically collected and displayed
• Chronological sorting – Always see the latest jobs first
• Time savings – Reduced job search time from hours to minutes
• Better insights – Visualize hiring trends and identify top employers
• Open-source solution – Built entirely with free R packages
• Scalable framework – Can be adapted to other job websites

---

📋 Future Enhancements

• Add email alerts for new jobs matching keywords
• Scrape individual job pages for more details (salary, location)
• Schedule daily automated scraping with cron
• Add job application tracking feature
• Integrate with job application database

---

📚 Lessons Learned

• Web scraping requires understanding HTML structure – Every website is different
• Respect website terms – Add delays between requests to be a good web citizen
• Handle errors gracefully – Websites change, so build robust error handling
• Clean data matters – Job titles need parsing to extract meaningful information
• Interactive dashboards make data useful – Visualizations help spot trends

---

🔗 Live Resources

📊 View Live Dashboard

📄 View Full Project Documentation

×

📈 Scalable M&E System for NGO Impact

Project Type: Digital Monitoring & Evaluation System

Status: 🟢 Live Demonstration Project

Tools & Technologies: KoBoToolbox, KoBo API, R, robotoolbox, tidyverse, dplyr, Shiny, shinyapps.io, GitHub

---

🧠 Problem Statement

Small NGOs and community-based organizations often rely on disconnected spreadsheets, manual reporting processes, and limited technical infrastructure. These challenges result in delayed reporting, duplicate records, poor data quality, and limited visibility into program performance.

---

🎯 Solution

I designed and deployed an automated Monitoring & Evaluation ecosystem that connects data collection, storage, processing, analysis, and reporting into a single reproducible workflow.

The system eliminates manual data downloads by pulling submissions directly from KoBoToolbox through its API, processing data in R, and displaying real-time insights through an interactive Shiny dashboard.

---

⚙️ System Architecture

• 📱 KoBoToolbox – Offline mobile data collection
• ☁️ KoBo API – Automated data retrieval
• 🧹 R + tidyverse – Data cleaning and transformation
• 📊 Scoring Algorithms – Vulnerability assessment
• 📈 Shiny Dashboard – Real-time monitoring
• 🚀 shinyapps.io – Cloud deployment

---

🗂️ Data Collection System

The project uses three integrated KoBoToolbox forms:

• Registry Form – Beneficiary enrollment
• Screening Form – Vulnerability assessment
• Intervention Tracking Form – Service delivery monitoring

Data entered through the forms is automatically synchronized and processed without requiring manual exports.

---

📊 Dashboard Outputs

• Active beneficiaries
• Vulnerability scoring
• Intervention coverage
• High-risk case identification
• Real-time monitoring indicators
• Automated reporting summaries

---

🚀 Impact

• Reduced reporting workload through automation
• Improved data quality using built-in validation
• Enabled real-time program monitoring
• Provided a low-cost M&E solution for NGOs and CBOs
• Demonstrated a scalable digital data ecosystem using open-source tools

---

🔗 Live Resources

📊 View Live Dashboard

📄 View Full Project Portfolio

×

📊 Labor Market Uncertainty and Employment Retention: "Job Hugging"

Project Type: Applied Labor Economics & Behavioral Data Science Case Study

Status: 🟡 Data Collection & Modeling in Progress | 🟢 Infrastructure Complete

Tools & Technologies: R (tidyverse, psych, stats, ggplot2), R Shiny, KoboToolbox API, World Bank & ILOSTAT APIs

---

🗺️ Production Artifacts & Live System Architecture

This pipeline is engineered as an automated data product. The data collection layout and operational tracking system are deployed up front to handle streaming field entries cleanly without manual intervention.

---

🧠 Problem Statement

Modern labor markets are experiencing a structural paradox: while widespread workplace dissatisfaction remains documented, voluntary job mobility is declining across multiple core industries. This study investigates this emerging behavioral pattern—empirically operationalized as "Job Hugging"—where workers choose to remain in their current roles despite actively considering alternative employment, driven by compounding macroeconomic anxiety and asymmetric household financial constraints.

---

🧪 Methodology & Measurement Instruments

This quantitative cross-sectional behavioral study integrates validated primary psychometric measurement sub-scales with structural secondary macroeconomic indicators:

• Job Satisfaction: Measured via the Brayfield-Rothe 5-item scale variables (js1 to js5).
• Job Insecurity: Tracked via the De Witte 4-item scale indicators (ji1 to ji4).
• Perceived Employability: Operationalized via the Rothwell & Arnold scale variables (pe1 to pe4).
• Financial Buffering: Modeled using numerical dependent ratios (income_dependents) and categorical timelines (emergency_buffer).

---

⚙️ Data Engineering & Econometric Framework

• Pipeline Automation: Configured programmatic API calls via robotoolbox to completely eliminate raw CSV exports, processing row binding algorithms live in workspace memory.
• Psychometric Validation: Integrated scale verification via the psych library to compute Cronbach's alpha coefficients, enforcing a strict internal reliability threshold of alpha ≥ 0.70.
• Model Specification: Implemented a Binary Logistic Regression Model estimating the log-odds probability of voluntary job mobility over a 12-month window against composite psychological scores and structural cross-country unemployment records.

---

🚀 Strategic Expected Impact

• Empirically bridges macro-level labor marketplace constraints with micro-level worker sentiment realities.
• Enables human resource professionals and policy researchers to cleanly differentiate between positive retention and security-driven talent lock-in.
• Deploys an automated, production-ready pipeline architecture highlighting an end-to-end data product lifecycle.

×

🛠️ validationcheck: Function-Based Data Validation Framework

Project Type: R Package for Data Quality Validation

Status: In Development

Tools: R, tidyverse, dplyr, testthat, GitHub

---

🧠 Problem Statement

Data validation in real-world monitoring and evaluation workflows is often fragmented, with analysts writing repetitive scripts for missing values, duplicates, invalid ranges, and logical inconsistencies. Existing tools such as pointblank provide powerful validation pipelines, but they require structured setup that can slow down rapid analysis in field-based environments.

---

🎯 Solution

validationcheck introduces a function-based validation system that allows users to define and execute validation rules in a simple, readable pipeline while tracking summaries, failure rates, and severity classifications (GOOD, WARNING, CRITICAL).

---

⚙️ Core Function: add_validation()

add_validation(
  agent,
  label,
  columns,
  rule,
  preconditions = NULL,
  check
)
      

---

📊 System Outputs

Severity Rules:
GOOD: 0% – 2% failure | WARNING: 2% – 20% failure | CRITICAL: > 20% failure

---

📦 View Package on GitHub →

×

📌 Strengthening Community Health Promoter Systems through Geospatial Mapping

Organization: Council of Imams and Preachers of Kenya (CIPK) Taita Taveta Branch – USAID TUJITEGEMEE Project

Type: Community Health Systems Strengthening & Geospatial Analysis

Tools: R, Shiny, Leaflet, KoboToolbox, dplyr, Excel

---

🧠 Problem Statement

During an OVC household validation exercise, program teams identified challenges related to the geographical distribution of households supported by Community Health Promoters (CHPs). Some CHPs were supporting households located far from their areas of operation, while others experienced overlapping household assignments. These inefficiencies affected case management, follow-up visits, and service delivery.

---

🧪 Methodology

• Designed and deployed a KoboToolbox survey to collect CHP geolocation data.
• Integrated household geolocation records collected during beneficiary validation exercises.
• Cleaned and processed data using R.
• Conducted geospatial analysis using Leaflet to map CHP and household locations.
• Identified overlapping coverage areas and household assignment gaps.
• Developed an interactive map to support evidence-based decision making.

---

🗺️ Project Workflow

---

📊 Key Findings

• Household assignments were not always aligned with the nearest CHP.
• Several households overlapped across CHP coverage areas.
• Significant variation existed in household distribution among CHPs.
• Distance between CHPs and households affected service delivery efficiency.

---

🚀 Outcome

• Improved visibility of CHP household coverage areas.
• Supported evidence-based household realignment decisions.
• Strengthened planning and resource allocation for OVC case management.
• Demonstrated the value of geospatial analytics for community health programming.